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ABSTRACT 

This is the final technical report (in three parts) 
of a 15-month long project to study the costs and value of library 
functions at five major research libraries. Twenty-one services or 
service aspects were studied, and numerous measures of the importance 
or benefit of the service to the users were made. These measures were 
studied together to lay a foundation for the development of an 
economically valid scale for assessing the impact of library 
services. More than 500 interviews were transcribed, and an empirical 
taxonomy was developed for classifying the contexts and the value to 
users of library services. Finally, the principles of data 
envelopment analysis have been adapted to this situation and 
illustrated using representative measures of library impact. This 
study has established not only the characteristics of the numerous 
services studied, but also the fact that a singl. uniform instrument 
can be applied to study many diverse services at many libraries. 
Instruments and a manual for their use in replicating these studies 
were also developed. The long-term- goal of this study is the 
development of a general taxonomy and metrology for library benefits, 
toward which substantial progress has been made, yet more remains to 
be done. Specific conceptual problems were identified that arise when 
the goal is to extract economically useful information from interview 
data, and points the way to further methods that will resolve these 
problems. Overall, it was discovered that users of library services: 
(1) value these services very highly; (2) have very little experience 
purchasing information services; (3) value the library more highly 
than other university-supplied services; (4) do not assign dollar 
estimate-; to the value of services which are commensurate with the 
cost of the services; and (5) can articulate, with proper 
questioning, the. context and purposes which bring them to use library 
services. Part 1 "Models, Methods, Results," summarizes the project's 
findings. Part 2, "Detailed History of Project Management and 
Processes," provides a detaileH history of project management and 
processes, including preliminary studies, selection of interviewers 
and research instruments, and delayed impact assessment. Part 3 is "\ 
Manual for Replication of These Studies." A fourth part, appendices 
A-H, include: instruments and SPSS codes; open ended coding; open 
ended coding results tables; cost data collection forms; interviewer 
training manual; miscellaneous forms; focus group moderator guide; 
and questionnaire instructions. (MAS) 
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EXECUTIVE SUMMARY 



This is the final technical report of a 15-month long project to study the costs and 
value of library functions at five major research libraries. A total of 21 services or service 
aspects were studied. Numerous measures of the importance or benefit of the service to the 
users were made. These measures were studied together to lay a foundation for the 
development of an economically valid scale for assessing the impact of library services. 
Further, more than 500 interviews were trarscribed, and through detailed content analysis an 
Empirical Taxonomy was developed for classifying the contexts and the value to users of 
library services. Cost estimates, using functional cost analysis, were developed for all the 
services. A Derived Taxonomy of Value in Using Library Service has been developed from 
the Empirical Taxonomy to provide a foundation for further research in this area. Finally, the 
principles of Data Envelopment Analysis have been adapted to this situation, and illustrated 
using i"epresentative measures of library impact. 

This study has definitely established not only the characteristics of the numerous 
services studied, but also the fact that a single uniform instrument can be applied to study 
many diverse services at many libraries. Additional goals met include the development of 
instruments and a manual for their use in replicating this study. The long-term goal of this 
research is the development of a general taxonomy and metrology for library benefits. 
Substantial progress has been made, and yet more remains to be done. The study has 
identified specific conceptual problems that arise when the goal is to extract economically 
useful information from interview data, and points the way to further methods that will 
resolve these problems. 

Overall, we have found that users of services at research libraries (1) value these 
services very highly, (2) have very little experience purchasing information services, (3) value 
the library more highly than other university-supplied services, (4) do not assign dollar 
estimates to the value of services which are commensurate with the cost of the services, and 
(5) can articulate, with proper questioning, the context and purposes which bring them to use 
library services. 
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1 COST AND VALUE OF SERVICES IN RESEARCH 
LIBRARIES: PROJECT OVERVIEW 

1.1 Problem, Objectives, Organization 

Research libraries and their supporting institutions are increasingly concerned about costs 
incurred and values received. This concern grows from difficult decisions on allocation of limited 
resources and the bewildering array of modern technology- based resources and services available 
to libraries. Libraries provide an ever increasing array of services. But the library is an internal 
service supported by the institution as a whole, in aid of its overall goals and objectives. 
Ultimately, the critical question that must be answered by institutional managers is: 

"Does the sum total of value flowing from the library justify our cost in maintaining it?" 

This most difficult problem leads, in turn, to many other ones. Studying the cost of 
university libraries raises a host of interesting methodological challenges. Adding the study of 
value, as perceived and/or defined by users, increases the methodological challenge many fold. 
These concern three aspects: the definition and determination (that is, the model and method) of 
the cost of library services; definition and determination of value of services to users; and finally 
combining in some way both cost and value, together. 

The goal of this study is to address the problem of developing models and methods for 
studying the cost and va'ue of library services m a way that can be pragmatically generalized and 
applied by libraries wishing to conduct similar studies. In other words, the goal of this study is 
to provide libraries and information services in general, and research libraries in particular, with 
methods for gathering information on the cost and value of their services; information that will 
aid in justification and decision making. 

The objectives are to: 

1. Define and apply methods for obtaining costs for several library services. 

2. Derive an empirical taxonomy of values for these services based on users' 
assessments. 

3. Provide methods for combining cost and value data. 

4. Provide a detailed description and manual that will allow for replication of these 
types of studies. 

To achieve these goals and objectives required an empirical study involving collection of 
a large amount of data from several libraries and a number • *" . Tfcrent services. Five libraries, 
21 services, and over 500 user interviews were involved, i/u. <t this among the largest, if not 
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the largest study of cost and value of library services. The study incorporated development of 
appropriate models and methods for study of cost and value, extensive validation of these with 
data from actual services and situations, and extensive documentation of these efforts. 

Accordingly, this Final Report is organized as follows: 

Part 1: Models, Methods, Results. 

Section L provides a general discussion of problems and issues and description of the 
approach to data collection. 

Section 2. discusses development of scales and provides statistics of the samples and 
scales involved. 

Section 3. is devoted to description and data on cost of different services. 

Section 4. is on value: description of approach to study of value and presentation of the 

Derived Taxonomy of Value in Using Library Service, together with some statistical 

results. 

Section 5. combines cost and value data. 

Section 6. provides directions for further study and application. 

Back matter, acknowledgements, a comprehensive bibliography, and title pages of other 
parts and appendices provide links to the remainder of this report, and to the literature. 

Part 2: Detailed History of Project Management 

Incorporates details on services studied; extensive statistics related to numbers in samples, 
interviews, and efforts to collect the data; and descriptions of development of 
instruments. This part describes what was actually done, the management of complex data 
collection, and the effort needed to get data. This material is for designers and managers 
of similar studies and any researchers seeking to replicate these results. 

Part 3.: A Manual for Replication of these Studies. 

Provides pragmatic suggestions for organization of a study of cost and value of library 
service, (detailed guidelines and .worksheets are in Appendix D). 

Appendix A. Provides the instruments used in interviews, and the mapping of variables defined 
for statistical manipulation into questions in questionnaires. 

Appendix B. Describes development of value taxonomies. Incorporates the initial coding of data 
and the first Empirical Taxonomy of Value derived directly from the user 
responses in interviews, and later used for development of the Derived Taxonomy. 

Appendix C. Gives the full statistical distribution of codes using the Empirical Taxonomy of 
Value, Version 1. 

Appendix D. Provides description and forms for cost data collection. 

Appendix E. A manual for training of interviewers. All data for determination of value were 
APLAB-B%D\CLR\FINAL.REV\FINALBIG.51 Copyright (c) 1 995 March 21, 1995 P. 8 
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collected through interviews. 

Appendix F. A variety of forms and letters used in the project, including letters used for 
recruiting, forms used for scheduling and record keeping. 

Appendix G. This is a guide for moderators of focus groups devoted to exploration of value of 
library services and libraries. 

Appendix H. Instructions for conducting of interviews, following questions in questionnaires. 
This complements and completes the defintion of the empricial taxonomy. 

In all, this Final Report incorporates extensive documentation, not only about the 
methodology and results, but also on specific procedures, instruments and forms used. We believe 
that such extensive documentation will be useful to those who would like to conduct similar 
studies, by using, adapting revising or incorporating parts of the procedures and instruments to 
their own needs. 

1.2 Non-Prof it Setting 

The critical question posed at the outset ("Does the sum total of value flowing from the 
library justify its cost?") is difficult enough to resolve in the corporate world, where there is at 
least one agreed upon overall measure of corporate progress: the net profits returned to 
shareholders. While this choice of a measure is subject to criticism from many directions, and 
is sometimes blamed for the failures of American industry to cany out long-term strategies aimed 
at developing market share, it does at least provide some starting point for the study of the role 
of libraries. The major research university, on the other hand is a quintessential example of a 
non-profit organization. Unlike hospitals, universities do not deal with a series of well-defined 
incidents or cases, which might be studied one by one to assess the impact of the library. 
Universities produce essentially intangible "products" such as "well educated students" and 
"cutting edge research". Thus, while financial soundness and fiscal responsibility are essential 
in the operation of a university, balancing the budget does not reflect progress towards those 
major goals. Even when a university broadens its goal statement to include a desire to have an 
impact on surrounding communities, and on the national cultural or economic profile, these 
enlarged goals also do not point the way to clear internal measures of progress. 



1.3 Scholarly and Practical Importance 

The question of value is attractive from a scholarly perspective. The issue from the 
scholarly perspective is "How can wc reasonably define and measure the contribution of the 
library (whose deliverable products are themselves intangible) to the overall goals and objectives 
of the university as a whole, whose goals and deliverables are even less tangible?". 

Aside from this scholarly interest there is a pressing practical reason to be concerned 
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about measuring the value of the library. The library has stood as a unique service organization 
in the university setting. Enshrined in a major campus building, it serves as a natural community 
focus for scholars, a point of pride for the ei.dre community, and its very bulk and contents have 
seemed in themselves to answer the question "Why do you cost so much?". All of this is being 
changed with the rapid development of computers and telecommunication systems which link the 
scholars and the students at their desks and laboratory benches to the metadocuments (catalogs, 
bibliographies, etc.) of the library on campus, of libraries around the world, and increasingly to 
the documents themselves maintained locally or elsewhere. With this in mind we have tried to 
include in our study examples of the "metadocumentary" services, such as enhancements to the 
online public access catalog, as well as services which provide access to information and to 
documents themselves. 

As the richness of a library will increasingly be measured by the range of materials which 
students and scholars can reach, rather than by the physical possession of those materials, it is 
essential that we develop, in the present setting, measures that assess, as well as we can, the 
value of present library services to the institution. With such measures in hand, libraries will be 
prepared to move through the transition, documenting as they go, that a decrease in the number 
of serials titles held or the number of bound volumes on the shelves, need not represent a 
decrease in the library's impact on the university. In fact, the best measures of impact would 
show the increase in value as access to materials is made more transparent for students and 
scholars. 

1.4 Measurement Focused on Patrons 

As a university does not conceive of itself in terms of manufacturing divisions, or product 
lines, there is no point in the institutional framework to which to address the question "How 
much does the library benefit the institution?". Rather, we believe, the correct point of inquiry 
is the individuals who make use of the services of the library. They generally do not conceive 
of their use of the library in terms of impact on the institution at large. Rather they sec 
themselves, at the moment of use, as engaged in some particular task or project whose goals are 
more or less in line with overall goals of the institution. 

Setting aside tasks and projects that are entirely personal in nature (hobbies, personal 
health concerns, preparation of tax returns, etc.) we can anticipate that students will primarily see 
their tasks in terms of completion of course work or research, in progressing towards an approved 
degree to be awarded by the university. Scholars on the other hand engage in a variety of tasks 
broadly defined as service or administrative tasks (such as verifying the credentials of a colleague 
who is being considered for promotion), teaching (which involves maintaining current awareness 
of pedagogical developments, as well as the preparation of specific lectures, assignments, 
exercises, laboratory projects, and so forth) and scholarship or research. 

In this situation we have decided that the most effective approach to assessing the value 
of the library is to focus attention on the specific task or project that brings the user to the 
library, and to ask questions about value, benefit and importance in the specific context of that 
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use. As we discuss at length in the body of the report, it is not an easy matter to ensure that 
respondents maintain this focus even during a brief ten-minute interview. 

1.5 Ideal Economic Perspective 

From the economic point of view one would build up the impact of the library on the 
institution by combining all of the instances in which the library contributes value to some task 
or project, weighting each such task or project in proportion to its own contribution to the goals 
and objectives of the university at large. This is, at present, only a broad conceptual structure, 
which can not be implemented. Universities have extreme difficulty in assigning any kind of 
relative weight or importance to their various missions, generally adopting (at least for the public) 
the stance that all components of the university mission are essential and therefore equally 
important. Similarly, it would be quite difficult to assess the importance of any single task or 
project as a contribution to any of the specific objectives of the university. However, even if this 
embedding of the results of a study such as ours into a larger economic picture of the institution 
can not be completed, we believe that there are techniques which will permit meaningful 
comparison of distinct modes of library operation in this setting. 

The idea behind our analysis is to take as a kind of constant or invariant the ability of the 
users of the library to assess the value of the library to their own tasks or projects. In other 
words, if the library, over a span of some ten years, is increasing its value to the tasks and 
projects of its users, then the natural presumption is that it is increasing its value to the 
institution. That assumption could only be questioned by a concrete demonstration that in some 
way the tasks and project of the users have evolved to be less in line with the goals of the 
university during the same period. Therefore, we believe that it is of importance to gain a deeper 
understanding of the value of the library to tasks and projects of the users. 

1 .6 The Nature of Library Value 

Where does this value lie? How, in fact, does the library aid those who use it? Broadly 
we conceptualize this in terms of three different activities which generally take place when a 
person uses a library: Acquisition , Cognition , and Application . Acquisition is the process of 
getting in touch with, touching, hearing, or seeing as appropriate, the devices or materials that 
bear the information sought. Cognition is the process by which the library patron "understands" 
the information in these materials and fits it somehow into his or her mental apparatus. Finally, 
Application occurs when the patron makes some use of this newly understood information. 

The application may consist of any action taken in consequence of the understanding of 
the information. That action may be a search for further information, because the present 
instance raised more questions than it answered, it may be a specific decision taken, an 
experiment designed, a policy introduced, etc. 

Working within this framework to define the value of the library for those who use it, and 
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by implication for the institution thai supports it, we have designed a study involving three 
instruments. 



1.7 The Costs of Services 

We study the costs of library services from the point of view of unit cost, or functional 
cost analysis. Unit cost analysis in production industries seeks to assess the total cost of 
producing a single object, such as a toothpick or a computer. The analogous concept for service 
industries involves defining specific units of service. Thus, the task is to assess the costs per 
service unit delivered. The specific point of view taken here does not address economic 
subtleties such as economies of scope or economies of scale. Rather it follows established 
accounting procedures that allocate all internal costs not directly associated with any particular 
service, to the range of services provided. Details, including a discussion of some special 
problems arising when one studies less than the entire institution, are given in Chapter 3. 

1.8 Scales for Assessing Value 

We have studied the value to users in terms of scale questions on an interview instrument. 
In these scale questions users are asked to represent their response with regard to something like 
the importance of a service on a 7-point scale. Such scales are generally recognized as having 
ordinal validity, but it is questionable whether they can be treated directly as interval scales of 
the type needed for most economic modeling. The specific set of questions that we used in the 
second wave of our study were refined through earlier preliminary studies, and still represent, in 
our judgment, a work in progress. Details of the questions themselves and of the potenual that 
they offer for the development of one or more abstract scales of library value are given in 
Chapter 2. Instruments are reproduced in Appendix A. 



1 .9 Development of Taxonomy 

The third aspect of our study is the collection and analysis of open-ended responses to 
questions about the tasks or projects that brought patrons to the library and about the value of 
the library for those tasks or projects. For the most part, this is aimed at developing a coherent 
taxonomy or classification of the kinds of value that users find in library services. We have 
found, in developing codes and categories for the analysis of these texts, that they form what is 
called a polyhierarchical structure. That is, even a brief response may reveal several different 
but overlapping aspects, each of which naturally falls into its own hierarchical structure. In 
general, we have limited ourselves to the assignment of no more than three specific codes to a 
brief (generally less than 100 words) answer given by a patron. The details of the development 
of this taxonomy, and of our study of the reliability of the codes thus defined are given in 
Chapter 4, and further in Appendices B and C. 
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1.10 Overview of the Libraries and Services Studied 



The study involved five large academic libraries in research oriented universities, thus the 
libraries can be considered both academic and research libraries. The universities participating 
in the study are shown in Table 1.1. 



Table 1 . 1 Participating Universities. 

• University of Maryland: A publicly supported research university with 
approximately 19,700 students. 

• New York University: A privately supported research university with 
approximately 21,400 students. 

• Princeton University: A privately supported research university with 
approximately 6,400 students. 

• Columbia University: A privately supported research university with 
approximately 14,700 students. 

• Rutgers University: A publicly supported research university with 
approximately 33,100 students. 



Number of students refers to full time students on the campus where the library is located. 

The universities above are not listed in any particular order. Later in the report, including 
in the table in this section we shall refer to libraries studied by number as Library 1 to 5., 
without reference to any given university. In other words, the numbering of libraries from 1 to 
5 does NOT correspond to the order of listed universities. We do this to preserve the anonymity 
of services studied. 

The study involved two time frames or periods during which interviews were conducted, 
respectively called Wave 1 and Wave 2. Wave 1 involved interviews with users immediately 
upon receipt of service. The interviews of Wave 1 were conducted during the Fall of 1993. In 
Wave t we concentrated upon value assessments which are perceived and articulated by users 
as soon as they finished receiving library service. After preliminary analysis of results from 
some of the libraries the instrument was revised and tested at additional services. The instrument 
was then elaborated for use in a set of interviews conducted in Spring 1994, which we call Wave 
2. Wave 2 sought expressions of the longer term value, which may be also perceived as impact, 
of the information gained and its relation to the task(s) of the individual and mission of the 
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institution as a whole. In this way we obtain both short and longer term assessments. The 
interviews of Wave 2 were conducted in two parts, for respondents who visited the library. The 
first part, which we informally labelled "audition", assessed immediate perceptions of value. The 
second part, administered by telephone, approximately two weeks later, obtained somewhat more 
long-term impressions. These were informally referred to as "callbacks". For respondents whose 
participation was solicited by mail or online, Wave 2 interviews consist of a single telephone 
interview approximately two weeks after the library use event. The details of data collected for 
each service and each of the Waves is given in Part 2. 

A general model for questionnaires was developed first, then adjusted to fit specific 
characteristic of each service studied. Thus, all the questionnaires for different services were 
specific to the service, but followed a general model that allowed for analysis of them all 
together. All the instruments are presented in Appendix A. 

Interviews were conducted by trained interviewers following the questionnaires and 
instructions. The manual for training of interviewers is in Appendix E, and instructions for 
interviewing in Appendix H. 

The figure below summarizes the services studied and methods used. 
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SERVICES STUDIED 



LIBRARY 1 



• Reference Services 

Information desk 
In-person reference 
Reference consultation 
© Materials Delivery Service 

LIBRARY 2 

• Art and architecture library - the collection 

• Biology library - electronic reference services 

• Psychology library 
LIBRARY 3 

• Enhanced online catalog 

- on-site 

- phone 

• Undergraduate reserve service 

- students 

- faculty 

• Science document delivery service 

• Carl Uncover pilot program 

LIBRARY 4 

• Electronic reference services 

• Music and media center 

- students 

- faculty 

• Interlibrary loan service 

LIBRARY 5 

• Patents service 

• Reference 

• Automated reference 



Exhibit 
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METHODOLOGY 

© On site intercept 

• Telephone 

- online volunteers 

- paper form volunteers 

- postcard volunteers 

SAMPLE 

• Faculty 

• Graduate Students 

• Undergraduate Students 

• Non-institution affiliated users 

Exhibit 1.2 



Table 1 .2 elaborates this, with very brief descriptions. More detailed descriptions of each service 
are given in Part 2 of this report, Section 3.2 The Codes given in the first column of this table 
are referenced again in Chapter 3, where we present the results of the Cost Analysis. 



Table 1.2. Table of Services Studied. 


Code 


Lib. Service 


Brief Description 


R5 


L1.S1.1 


Information Desk 


R8 


L1.S1.2 


Reference Desk, Branch 1 


R4 


L1.S1.3 


Reference Desk, Branch 2 


R7 


L1.S1.4 


Reference Consultation Service 


Dl 


L1.S2 


Materials Delivery Service (MDS) 


LI 


L2.S1 


Art and Archeology Library 


R6 


L2.S2 


Biology Library - Electronic Reference 


R9 


L2.S3 


Psychology Library - Reference 


CI 


L3.S1 


Enhanced Online Catalog Service 


C2 


L3.S1.V2 


Remote Users 


V3 


L3.S2 


Undergraduate Reserve Service 


V4 


L3.S2.V2 


Undergraduate Reserve Service: Faculty 
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Table 1.2. Table of Services Studied. 


Code 


Lib. Service 


Brief Description 


NotCosted 


L3.S3.1 


Document Delivery Service 


D2 


L3.S3.2 


Carl UNCOVER 


R2 


L4.S1 


Electronic Reference Resources 


XI 


L4.S2 


Interlibrary Loan Service 


VI 


L4.S3 


Music and Media Center 


V2 


L4.S3.V2 


Music and Media Center: Faculty 


R3 


L5.S1 


Automated Reference Service 


Rl 


L5.S2 


Traditional Reference Service 


L2 


L5.S3 


Patents Service 



1.11 Replicability of this Study 



The process in this study was a complex one, involving interviews with over 500 
library users on site at the library, by telephone, and in two-part interviews whose second part 
was a follow up some two to three weeks after the use of the library. We believe that the 
methods themselves will be of considerable interest to library managers and to other scholars 
as well. The complete details of the process, given at a level that will enable such 
organizations to replicate or adapt our process are given in Parts 2 and 3 of this Report, and 
in a series of appendices. 

1.12 Combining Values and Costs 

The overall goal of this study is to relate value to costs. As we have said, the 
measures of value are not natural economic measures in themselves. We have, therefore, 
turned to a relatively new technique for the analysis of non-profit service activities, called 
Data Envelopment Analysis. The details of this, and of our efforts to link together the 
impacts and the costs are given in Chapter 5. 

Finally, in Chapter 6, we consider the implications of the present study both for library 
practice and management, and for further research. 
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2 STATISTICAL RESULTS AND SCALE DEVELOPMENT 



2:1 Scale Variables and Metrology 

All of our survey instruments contained many binary or scaled questions. On the one 
hand, these questions permit us to report the profile of user responses. These responses are of 
interest in themselves, as they tell us things like the frequency with which the library is rated 
very highly, the experience that patrons have with purchasing information (extremely low) 
and so forth. In addition, standard techniques from the social sciences often make it possible 
to extract one or more underlying factors which can be used to develop a scale for 
measurement of some abstrac . concept. 

In this case the abstract concept of interest is "the importance of the library in the 
context of the user's task or problem". To expose this factor we included a number of 
questions specifically asking for the importance and for other variables that we thought might 
clarify the concept such as "Was what you got from the library worth the time that it took to 
get it?" and "How confident are you in the information that you received from the library?". 
A good metric scheme or metrology should extract, from the data, a measure of the impact on 
the users which distinguishes among services and service events, and bears some sensible 
relation to the costs of providing the service. 

2.2 Reliability Analysis 

The set of such questions, shown in Section 2.6.3 were analyzed in two ways. One 
makes use of a technique called reliability analysis which explores the extent to which the set 
of questions together can be used additively to define a single measure of some underlying 
concept. The corresponding statistic is a reliability co-efficient called Cronbach's Alpha. The 
results of this analysis were discouraging, as the overall reliability of the set never exceeded 
60%, and there were no single items whose removal produced a very substantial improvement 
in that score. 

2.3 factor Analysis 

We then considered the possibility that the set of questions were driven by two or 
more underlying factors which must be resolved in order to develop a scale. This was studied 
using factor analysis, with a technique called principal axis factoring (which takes into 
account the fact that every one of the variables may be composed of both a contribution due 
to underlying factors and a contribution due to its own idiosyncratic variation) and a 
technique for "rotating the factors" called oblique minimization which allows for the 
possibility that the underlying factors are not totally independent of each other ("orthogonal") 
but may have some interaction with each other. 

The results of this analysis were initially encouraging, as they revealed two underlying 
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factors. One, driving two of the variables (low difficulty in use and high confidence in the 
information) does not seem directly related to our goal of measuring the impact. However, it 
is interesting to note that confidence in the information, which one would think of as a highly 
cognitive variable strongly dependent on the task and problem context, is most closely related 
to a simple variable, lack of difficulty in use, related to the acquisition process itself. (For 
technical reasons since the question was coded in terms of increasing difficulty, a surrogate 
variable obtained by subtracting the difficulty score from 8 was used to represent the lack of 
difficulty). 

The remaining factor contributed heavily to the variables that seem to us most likely 
to measure the value of the library service. In fact, had we been able to terminate our 
analysis at this point, the results would be quite positive. However, our goal is to develop a 
bi-variate representation in which the impact and the costs are considered jointly. Such a 
representation will not be very interesting unless the levels of impact associated with the 
different services (recall that we are only able to assign a cost to a unit of service overall, and 
can not assign a cost to the individual instances of use) can be distinguished in terms of the 
impact measure. 

2.4 Variation of Value with Service 

With this in mind, we undertook a simple analysis of variance on the factor score 
which seems to underlie the several measures representing value. The purpose of an analysis 
of variance is to determine whether the differences between the average levels of a variable, 
for the several different services, are relatively large compared to the amount of variation of 
that variable within any particular service. The results here were disappointing, as there was 
no significant difference among the average levels of this "impact factor score" among the 
various services studied. It is for this reason that our analysis using data envelopment 
techniques (see Chapter 5) is at this point prospective and theoretical rather than 
determinative of the relationships of interest. 

Of course, any negative result immediately raises a research challenge. We have 
found that the set of questions that we use does not resolve the levels of impact in a 
distinguishable way. One possibility (the totally pessimistic view) is that it is not possible by 
any means to resolve those levels of impact. Setting this aside, we ask what aspects of the 
present situation may have led to the negative results that we find here even if the concept of 
impact is indeed measurable. 

2.5 Confounding Factors: The Local Context 

We believe that the outstanding confounding or limiting factor is the fact that each 
interview was conducted in a way which provides an overwhelmingly local context. That is, 
the patron is directed (as in all techniques related to the critical incident method) to focus on 
a specific use of the library. The benefit of this is well known, as it eliminates halo effects 
and decreases the chance that the patron is simply telling us what she or he thinks about the 
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library in general. The negative consequence, which causes problems for the present research, 
is that that same isolation and focus inhibits any tendency that the patron might have to 
mentally compare this particular instance of library service with other instances in her or his 
experience. Because of that, we do not have very much prospect of obtaining a globally 
calibrated metric result. To put it in simplest terms, if all I wanted from the library this 
particular day was to photocopy an article and I managed to do so, I may be extremely 
pleased. That this article is of very little importance to me is something which is not 
addressed by the set of questions that we have framed, and formulating questions for this is a 
challenge which we will take up in Chapter 6. 

The remainder of this chapter is a detailed exposition of the most interesting of the 
statistical results that we have found. 

2.6 Detailed Statistical Results 

2.6.1 Descriptive and Demographic Variables. 

The results for each varaiable are presnted in tabular form, showing the 
groups, the internal "Value" repesenting that group, the frequency, and the 
frequency represented as a Percentage of all cases or of all valid cases. 



2.6.1.1 AGEGRP 


Aqe of 


interviewee 




Value Label 




Value 


Frequency 


Percent 






.00 


2 


. 4 


Under 18 




1 . 00 


3 


. 6 


18-25 




2 . 00 


172 


32 . 6 


26-29 




3 .00 


119 


22 . 6 


Thirties 




4.00 


120 


22 . 8 


Forties 




5 . 00 


73 


13 . 9 


Fifties 




6.00 


26 


4.9 


Sixties or above 




7 . 00 


12 


2 . 3 








1 


Missing 






Total 


528 


100 . 0 


Valid cases 527 


Missing cases 1 




The preponderance 


of the 


respondents are under 


thirty. 


2.6.1.2 AREAST 


Area of 


study 














Valid 


Value Label 




Value 


Frequency 


Percent 


Humanities 




1 . 00 


142 


27. 5 


Social science 




2 . 00 


96 


18.6 


Natural science 




3 . 00 


83 


16.1 


Engineering 




4.00 


14 


2.7 


Professions 




5 . 00 


142 


27 . 5 


Other 




6. 00 


33 


6.4 






88.00 


7 


1.4 








1 


Missing 


No response 




-1.00 


10 


Missing 






Total 


528 


100 . 0 



A complete list of the specific disciplines assigned to each of these groups 
is given in the manual. 
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3 BEST COPY AVAILABLE 



'2 6 1 3 FRF^FR How 


frequent ly do 


you use the 


service 








Valio. 


Value Label 


Value 


Frequency 


Percent 


No response 


.00 


2 


. 5 


Less than a month 


1 . 00 


74 


19 . 1 


Once a month 


2.00 


49 


12 . 6 


2-3 times a month 


3 .00 


97 


25 . 0 


4 timet a month 


4.00 


39 


10 . 1 


More than 4 a month 


5.00 


127 


32.7 






140 


Missing 




Total 


528 


100 . 0 



This question was introduced early in the first version of the instrument, and 
was later dropped because we found that it framed the interview in terms of 
the service, making it very difficult for respondents to address the key 
issues of context and purpose. 



2.6.1.4 GENDER 



Value Label 



Sex of interviewee 



Value Frequency 



Male 
Female 



1 . 00 
2 . 00 

Total 



258 
270 



528 



Valid 
Percent 

48.9 
51. 1 

10C . 0 



The sexes are quite equally represented in this sample, suggesting that they 
are quite equally represented among the users of the library. It would be of 
interest to compare this with the distribution of students, by sex, at each of 
the institutions. 



2.6.1.5 LIBRY 


Library 






Value Label 




Frequency 


Percent 


U. of Maryland 




112 


21.3 


NYU 




95 


18. 0 


Princeton 




115 


21.8 


Columbia 




110 


20 . 9 


Rutgers 




95 


18. 0 






1 


Missing 




Total 


528 


100 . 0 



Valid 



The study obtained nearly equal numbers of respondents from each of the 
institutions . 



2.6.1.6 STATUS 


Patron cateqory 












Valid 


Value Label 


Value 


Frequency 


Percent 


Fac/Sta 


1. 00 


92 


17 .4 


Graduates 


2. 00 


334 


63 .3 


Undergraduates 


3 . 00 


91 


17.2 


Others 


4.00 


11 


2.1 




Total 


528 


100. 0 



Although we would have liked to obtain more faculty input, faculty are not as 
well represented among the users of the library as are graduate students. 
Undergraduates were interviewed in cases (such as reserve service) where they 
are the primary users of the service, and where there was too little traffic 
to meet sample quotas without including them. 
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2.6.2 About the Impact of the Service 



Name of Variable 



Distribution of Responses 
Label 



Cases Yes 



No 



ALTWAY Other alternatives 

HURTFL Did the service hurt 

IMPACT Impact or no impact to your project 



386 79% 20? 

12 17% 835 
134 90% 105 



A substantial fraction of the respondents felt that there was another way that 
they could get the service which they obtained from the librar-'. Very few 
reported that the service was in any way hurtful to their activities. This 
was a conditional question, only addressed to those who assigned a very lew 
score to the benefit derived form the service. Finally, 120 of the 134 to 
whom the question was addressed, reported that the service. did have an impact 
on the project which brought them to the library. 



ERIC 
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3 COSTS OF LIBRARY FUNCTIONS AND SERVICES 



3.1 General Principles 

Our determination of library costs follows general principles for unit cost accounting. 
The application of these to the library setting has been set out in considerable detail elsewhere 
[Kantor 1984, 1986c, 1989a] and will not be reviewed here. It is not difficult to identify the 
direct costs associated with a particular service, such as the supplies consumed or the labor of 
people who work on no other service. It is somewhat more difficult when individuals work 
on several services, but well established techniques can be used to allocate their time among 
those services. Essentially, there are two difficult problems in the cost allocation: allocation 
of salaries, and allocation of the cost of shared resources. The techniques used in this study 
are adapted, with permission, from the Tantalus Inc. FUNCOST software for library cost 
analysis. Details of the data collection are given in the Appendices. 

3.1.1 Methods for Cost Analysis: General Principles 

To begin, we situate our model for cost, analysis in the general framework of economic 
analysis. There are several ways in which costs can be divided, and we have necessarily 
made some choices with regard to each of these options. 

To begin with, costs can be divided in to start-up costs and ongoing costs. In this 
project all services had been established some time before the analysis was done, and only 
ongoing costs are considered. Costs are further divided into fixed costs and variable costs. 
The fixed costs are those costs necessary to provide the service at all. For example, a 
circulating collection must contain some books. The variable costs are those costs which are 
approximately proportional to the volume of services. In the accounting model of cost 
analysis used here, the average costs are reported. This is similar to treating all costs as 
variable costs. 



Thus the unit costs reported here, while they provide a full account of expenditures, 
cannot be used as reliable predictors of cost change for all other situations. For example, if 
the volume of service increases moderately, the corresponding increase of cost, at the same 
level of service quality, will probably be less than the figure reported here. If, however, the 
present system is operating close to capacity, an increase in volume is likely to produce a 
noticeable decrease in service quality, unless there is some compensating investment in 
equipment, personnel, or both. Such a quantum jump in capability would result in an increase 
in the average cost of service. 

Economies of scale, which may be either positive or negative, represent deviations 
from the straight-line dependence of variable costs on volume of service. For example, with 
positive economies of scale, the average unit cost decreases as the volume of service 
increases at constant service quality. Similarly, with positive economies of scope, the total 
cost of providing several different types of service will be less than the sum of the costs of 
providing them separately. It seems reasonable to suppose that economies of both scale and 
scope are present in library operations, and in the electronic versions of information service 
that arc an increasing lactor in library operations. However, this study is, by design, limited 
to consideration of services in isolation, and at a few libraries. Thus it is not possible to 
model either economies of scale or economies of scope. 
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To sum up, our model calculates, for each service, the average unit cost of service at 
the present level of operation. All shared resources are pro-rated, using appropriate methods, 
to the one service under consideration, and to "all other services". This method probably 
under-estimates the cost of providing any particular service on a stand-alone basis. This 
shortcoming is not of interest to the community of research libraries, who are the primary 
audience for this study, as they do not offer services in isolation. 

Costs of equipment are normally dealt with by reference to depreciation tables, or 
amortization schemes, which are, in turn, developed by industry in respond to the tax laws. 
Since all of the institutions studied are non-profit, this information is not available. We have 
assumed throughout that computer equipment is to be expensed over a three or a five year 
period, depending on the likely reasonable life. We note that the reasonable life of computer 
equipment today is determined more by considerations of obsolescence than by considerations 
of maintenance and useful working life. 

3.1.2 Methods for Cost Analysis: Worked Examp le 

The principles of analysis are illustrated here for a single worker, and for a single 
piece of shared equipment. For full analysis this procedure is applied in turn to each worker, 
and to each shared resource, and the resulting costs are summed. 

3.1.2.1 Analysis of salary expenditure distribution 



We suppose that a particular worker "ABC", with annual salary $30,000 has reported 
the following distribution of time over several activities: 



Name 


Salary 


Direct 


Support 


Other 


Administr 


Prof. 






Service 


Service 


Service 


ation 


Devpmnt 


ABC 


30,000 


10 


12 


12 


8 


3 J 



The distribution reporting figures may be percentages, or they may be hours per day, 
hours per week, or we~\s per year. The analysis utilizes only the ratios among these 
distribution figures, so that the choice of a base does not matter, and there is no need to 
require that the distribution figures total to 100%, or to a specified number of hours per day 
or per week. 



In this analysis "Direct service" refers to the service whose cost is to be calculated. 
"Support service" refers to one or .nore activities which contribute solely to this visible 
service. For example, if the direct service is circulation, reshelving is a support service. If 
the direct service is an online catalog, programming is a support service, and so on. The 
first step in the analysis is to combine the support service into the direct service. The results 
is: 



Name 


Salary 


Direct 


Support 


Other 


Administr 


Prof. 






Service 


Service 


Service 


ation 


Devpmnt 


ABC 


30,000 


10 


12 


12 


8 


3 


ABC 


30,000 


22 




12 


8 


3 
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The next step is to simply remove the figure related to professional development. This . 
amounts to increasing the cost of all other activities as if that share had been distributed over 
it. In other words, the cost per unit of this person's work on end-products is increased. 



Name 


Salary 


Direct 
Service 


Support 
Service 


Other 
Service 


Administr 
ation 


Prof. 
Devpmnt 


ABC 


30,000 


10 


12 


12 


8 


3 


ABC 


30,000 


22 




12 


8 


3 


ABC 


30,000 


22 




12 


8 





Next, the administrative cost is distributed over the services themselves. In the most 
complex case, there may differing proportions to apply, in which case Administration must be 
treated as another form of support service. In our analyses this appeared necessary at one 
library. However, as it was later decided to eliminate administrative costs from all 
calculations, the details will not be given here. Thus, for practical puiposes, the calculation 
of labor cost concludes with the distribution of the annual salary, in the proportions indicated 
here. That is, we do not "burden" the reported costs with administrative costs. For example: 
the cost assigned to the Direct Service of interest is: 

3(),000*(22/(22+l 2+8)=30,000*22/42= 1 5 ,7 1 4. 



Numbers are rounded to the nearest whole dollar, so that the results may not sum 
exactly to 30,000. They should, however, be within a dollar or two. 



Name 


Salary 


Direct 
Service 


Support 
Service 


Other 
Service 


Administr 
ation 


Prof. 
Devpmnt 


ABC 


30,000 


10 


12 


12 


8 


3 


ABC 


30,000 


22 




12 


8 


3 


ABC 


30,000 


22 




12 


8 




ABC 


30,000 


15,714 




8,571 


5,714 





3.1.2.2 Analysis of equipment and other expenditure distribution 



The treatment of costs of equipment and other shared resources is much the same. For 
example, for a central computer with a purchase cost of 40,000 and an annual maintenance of 
$5,000 we assign an annual cost based on a five-year lifetime: 

Annual cost = 40,000/5 years + 5,0()0/year= 

=13,0()()/ycar. 

Wc then distribute this cost according to some reasonable measure of the use of the 
computer. Suitable candidates for this measure include: the number of ports dedicated to a 
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service; the number of CPU cycles (or CPU seconds) consumed by that service, or the 
number of terminals provided for the public to access that service. In all of the services 
studies here, the number of terminals was used as the basis for cost assignment. 



A typical calculation looks like the following: 



Equipmt 


AnnCost 


Direct 
Service 


Support 
Service 


Other 
(1) 


Other 
(2) 


Other 
(3)... 


Server 1 


13,000 


5 


1 


3 


2 


2 


The distribution of the cost to the direct service of interest is then 
13,000*5/(5+1+3+2+2) = 13,000*5/(13) = 5,000. 


Equipmt 


AnnCost 


Direct 
Service 


Support 
Service 


Other 
(1) 


Other 

(2) 


Other 
(3)... 


Server 1 


13,000 


5 


1 


3 


2 


2 


Serverl 


13,000 


5,000 


1,000 


3,000 


2,000 


2,000 



3.1.2.3 Example Calculation of a Unit Cost of Service 



Finally, the costs of all labor, and of all equipment (of which only the two 
representatives just calculated are shown here) are combined and divided by the annual 
number of service events. This number may be either the result of a complete tally, or an 
estimate based upon a typical week (or month) or some other sampling study of the use of the 
service. In this example we suppose that there are 15,000 service events per year. 



Origins 


Value 


Labor Cost 


15,714 


Computer Serverl 


5,000 


Total Cost 


20,714 


Service Volume 


15,000 


Unit Cost(Average) 


$1.38 



3.2 Administrative Costs 



Most university libraries engage in some degree of collegial management, with the 
result that almost all workers spend a certain amount of their time in committee meetings and 
related activities having to do with the administration and management of the library. The 
problem in general is how to allocate that time to the specific activities. When the library as 
a whole is being considered one may argue that all of that time should simply be allocated 
against all of the identifiable services. It is typically allocated in proportion to the total salary 
expense of the services, in accordance with established customs for the accounting of 
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' administrative overheads. 



However, in this particular study we did not analyze all of the services and functions 
provided by each participating library. This means that we do not know the correct base on 
which to allocate the administrative time, particularly committee work, among the several 
services. After some efforts to reconstruct this data by extended conversations with the 
libraries involved we came to the conclusion that it would not be possible to develop 
uniformly reliable and comparable measures across the several libraries. Examples of the 
problems are (1) library administrators who are involved in supervising multiple services and 
who would have to allocate their time against all of those services, requiring a much more 
detailed cost study than has been undertaken here and (2) library professionals who spend a 
significant amount of time in committee meetings but have no natural basis for allocating that 
committee time among the several tasks on which they spend their time. 

In view of these difficulties we have sought to strip away the administrative 
component of the costs as they are reported for each specific service. Technically, this has 
been done by counting all administrative time as if it were another identifiable service 
activity. In other words, it is as if we said that the library provides reference services, access 
services, administrative services and so forth to itself. In fact, we know that these internal 
services simply serve to add value to the specific deliverable services. 

We believe, based on our examination of the raw data, that there are substantial 
variations to be observed in the ratio of administrative costs for a service to the other direct 
costs of the service. We are not. in a position to separate that variation into its two most 
natural components: variation due to the nature of the service being managed, and variation 
due to the nature of management procedures at the library being studied. These issues are of 
enormous importance in connection with the total quality movement in library studies, but are 
beyond the scope of the present research. 

3.3 Costs of the Collection 

In addition to the problem of administrative costs there ls the enormous problem of 
how to deal with the cost of the collection, and of the organization of that collection, in 
determining the functional or unit costs. In a full scale study of a library this is done by 
forming all of those costs into an overhead pool and allocating them among several services 
in proportion to the degree to which those services make use of the books and other materials 
in the collection. Jn this case we do not have the broad scope of information necessary to 
complete this. For most of the services it does not seem to be an issue. For reference 
services we are therefore not including the costs of the specific collection of reference 
materials maintained, and at the same time setting aside any consideration of the use that 
reference service makes of the collection as a whole. 

However, for one of our specific studies it is not possible to sidestep this question. 
That is the use of the non-circulating collection at an Art & Architecture Library. In this case 
we have accepted the annual book purchase cost, and the assigned share of central processing 
costs, as costs of the service provided. In the same spirit, we have included the cost of the 
reserve materials at an undergraduate reading room. However, it was not feasible to use the 
same approach in determining the costs of the audio-visual center at a different library. 
Similarly, the cost of developing the catalog records, which are accessed by the online 
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system, have not been included. 

Fringe costs were systematically excluded, to make the comparison more uniform. We 
did, however, use the actual labor costs. There are variations in wage scale, as two of the 
libraries are in New York City, two are in New Jersey within 50 miles of New York, and one 
is in suburban Washington DC. 

3.4 Summary of the Cost Data 

Bearing these caveats in mind, we now turn to a summary of the costs of the specific 
services. To assure the confidentiality of the libraries, we have separated the numbering of 
the libraries from the naming of the libraries, as described earlier. 

Table 3.1 Reported Costs of Services, labelled by general category of service. Services 
are presented in increasing order of unit cost. {D-Materials Delivery. C-Catalog or 
online service. R-Reference Service. V-Reserve Collection. L-Entire Collection. 
X-Other service. See more detailed descriptions of the services in Volume 2: Process.) 



Cost Code Lib. Service Brief Description 



N/A 


XX 


L3 . 


S3 


1 


Document Delivery Service 


0. 


70 


CI 


L3 . 


SI 




Enhanced Online Catalog Service 


0 


70 


C2 


L3 . 


SI 


V2 


Remote Users 


1 


16 


Rl 


L5 


S2 




Traditional Reference Service 


1 


88 


Dl 


LI 


S2 




Materials Delivery Service (MDS) 


2 


00 


R2 


L4 


SI 




Electronic Reference Resources 


2 


65 


V2 


L4 


S3 


V2 


Music and Media Center.: Faculty 


2 


65 


VI 


L4 


S3 




Music and Media Center 


6 


12 


LI 


L2 


SI 




Art and Archeology Library 


6 


87 


R3 


L5 


SI 




Automated Reference Service 


8 


18 


R4 


LI 


SI 


.3 


Reference Desk, Branch 2 


8 


59 


V3 


L3 


S2 




Undergraduate Reserve Service 


8 


59 


V4 


L3 


S2 


.V2 


Undergraduate Reserve Service: Faculty 


12 


30 


XI 


L4 


S2 




Interlibrary Loan Service 


14 


. 62 


L2 


L5 


.S3 




Patents 


18 


.35 


R5 


LI 


. SI 


. 1 


Information Desk 


18 


.80 


R7 


LI 


.SI 


.4 


Reference Consultation Service 


20 


.35 


R6 


L2 


• S2 




Bioloc^ Library - Electronic Reference 


33 


.36 


R8 


LI 


. SI 


.2 


Reference Desk, Branch 1 


35 


.52 


R9 


L2 


.S3 




Psychology Library - Reference 


36 


. 13 


D2 


L3 


.S3 


.2 


Carl UNCOVER 



We see (not surprisingly) that costs range from a low of less than a dollar, typically 
for the use of some computer-supported service that is heavily trafficked (and for which we 
have not included the costs of developing the underlying database), to a high of over thirty 
dollars, for some classes of reference and materials delivery services. These figures must be 
regarded as approximate, since there are variations in the degree to which certain 
administrative costs have been included, there are arbitrary decisions made in distributing the 
cost of shared equipment, and they do not include fringe costs. However, we are confident 
that they accurately represent the range of costs for the services considered. 
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We note that the range of costs for reference services is enormous: from a low of 
$1.60 to a high of 35.52. The low end is surprising, while the high end is consistent with the 
results of other detailed studies of reference costs in the context of full library studies. This 
figure is, presumably, completely unsuspected by patrons who assign a value of $5-$ 10 to a 
library service event. Of course the average cost of this service is hard to control, since the 
library sets out the service, but the patrons do or do not make use of it. 

Similarly, the two materials delivery services vary enormously in cost. Interestingly, 
the two cases in which we studied an entire collection fall into the middle of the range, with 
the specialized patent operation being somewhat more expensive than the (equally specialized) 
art and architecture setting. As has been determined in other studies, interlibrary loan service 
costs far less than the cost of buying and processing a book, although we note that this 
includes only the costs experienced by the borrowing library. 

More detailed confidential reports have been prepared for the participating libraries, 
and those will permit further discussion of the accuracy and comparability of the numbers 
presented here. Ultimately the participating libraries, which have been assured confidentiality 
with regard to cost and performance data may or may not decide to meet and share that 
information. 
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4 DEVELOPING A TAXONOMY OF VALUES FOR 

LIBRARY SERVICES 



4.1 Framework: assumptions, definitions, approach 

In seeking and using a library service a user engages in an interaction with a library. 
That interaction may be studied and characterized from a variety of perspectives - individual, 
social, technological, economic and so on. While traditionally the interaction involves direct 
physical proximity to the library, with the advent of telecommunication and computer 
technologies the interaction can also be remote. 

In this part of the project we have concentrated on a study of the reported perceptions 
of the value that individuals derive from interaction with a variety of library services. The 
objective is to derive a taxonomy of values of library services as perceived, identified, and 
assessed by users. As such, it is not based on a prior theory of value. In so far as the data we 
have gathered lead to a coherent picture, that picture can be said to represent a "grounded 
theory" whose validation must depend on further applications of it. 

Given the scope of this study, the data from which our taxonomy is built reflects the 
concerns and language of users of academic libraries. It is useful therefore to distinguish 
between the "empirical taxonomy" (ET) as set out in Appendix B, and the "derived 
taxonomy" (DT) explicated in this portion of the report. We believe that the structure and 
broad classes of the DT are valid for many kinds of libraries, and information services, while 
the specifics of the ET will be adjusted (deleted, added, redefined) as appropriate to diverse 
services. In addition we go into considerable detail about the process of deriving and testing 
the taxonomy because we believe that it can be replicated and applied by others interested in 
the issues of library economics. 

We review the three premises of this analysis: 

1. Users interact with a library service, that is, use or attempt to use a library 
service, for a given reason or reas—,. While during the interaction the 
reason(s) may be altered or changed, they are still present at all times. 

2. As an outcome of the interaction, users obtain responses or results, bo they 
positive or negative. 

3. Users evaluate or assess the interaction and the responses or results in relation 
to their reason(s) for using the library service(s). 

In assessment of value, the "reasons and interaction" may be considered as a cause and 
"results or responses" as an effect. When examined in more detail, however, both of these 
are complex and may involve a number of distinct conceptual variables. 

The value of any library service is then assessed, using judgment by users (or user 
: : rrogates), about the qualities of the interaction with the service and the benefits, or worth of 
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the results of the interaction. Ideally, this is assessed, by the user, in relation to the reasons 
for using the service. A complete value assessment would establish a relation among: the 
reasons for using a library service, the processes in interaction and thr results or responses 
obtained. 

With this conceptual framework, our questions to the users were addressed to elicit, 
from the users, the following facets: 



I. 



Reasons for use, 



2. Qualities of interaction (use) related to a given service, and 

3. Worth, benefits or implications of subsequent results from the use. 

Results may be considered the "product" of interaction with a service. As in other 
service situations, the value to the user is a composite of both the service and the product and 
not just the product alone. This is a principle of assessing quality in Total Quality 
Management (TQM) where, for example, the accuracy of support services is assessed jointly 
with service characteristics such as cycle time or responsiveness. 



4.2 Related works and concepts 

Libraries' information services have been evaluated or assessed from a number of 
perspectives. Of concern here is evaluation from the perspective of users. There is a sizable 
literature on the need for "user-centered evaluation" (Bawden, 1991). Such studies have been 
conducted using a variety of measures, such as: satisfaction, success, utility, relevance, 
completeness, precision, timeliness, accuracy, impact and many others (Baker & Lancaster, 
1991, Kan tor 1984). Many such studies are based in one way or another on some 
assessment involving user judgement of value. 

There is also a substantial literature on the notion that library processes "add value" 
(Taylor 1986). But the value itself, while much written about, is rarely studied with suppoit 
of empirical evidence. While there is a place for theoretical, logical and philosophical 
reasoning (e.g. Taylor, 1986) there is a need to develop a substantial body of reproducible 
empirical evidence. 

One of the few studies of value related to an information service which involved 
empirical evidence was an assessment of the impact of using MEDLINE (Wilson et al. 1989). 
Using a critical incident technique in surveying a large number of MEDLINE users, the 
analysis resulted in three taxonomies: reasons why individuals needed information from 
MEDLINE, reasons why individuals chose to do a MEDLINE search (rather than use other 
means of obtaining information), and impact of the information obtained from MEDLINE on 
medical decision making. 

Finally, a recent set of on value and impact of information studies, sponsored by the 
British Library Research, addresses many of the issues treated here, but in addition , a 
number of policy issues are treated as well (Feeney & Grieves, 1994). 

The taxonomies were very specific, because they all evolved around a single 
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discipline, medicine, and a single activity, medical decision making. As yet, no comparable 
taxonomy has been attempted involving values appropriate to a multitude of library services, 
for a multitude of reasons, and a number of Jisciplines, a* we do here. In this sense the 
present work is a pioneering effort, with all the implied limitations. 

Related to study of the value of library services is the study of value of information in 
economics and management science. A review is given by Repo (1989). There a distinction is 
made between exchange values which involve information resources, products and services in 
a framework of the information market, and value-in-use in a framework of use of 
information by individuals and their information tasks. Exchange values can be studied by 
exploring the market for services, following the tools of economics. 

However, value-in-use involves cognitive aspects and has to be studied using primarily 
cognitive, problem-solving, and decision-making concepts. In a sense, this is an example of 
the economic theory of the value of information. However, that framework is most developed 
when the expected utility theory of value can be applied and, at the present time, we do not 
know how to do that in the library setting. So far the two, exchange values and value-in-use. 
have not been successfully studied together. After reviewing many theoretical approaches, 
Repo concludes: 

It seems obvious that there is not a single theory which fully explains the value of 
information. It does not seem probable that one can develop such a theory easily. This 
is because the empirical fact that individuals give different values for the same 
information depending on context. 

He suggests a dual approach to [study of] value of information: 

1. The exchange value of information products (service, channel, system) should 
be studied using "classical" economic methods. 

2. The value-ir.-use of information should be studied using the cognitive approach 
which takes the user, the use and the effects of the use of information into 
consideration. 

In the present project we are accomplishing steps towards this synthesis, by applying 
accounting principles to establish a kind of market value of services, and using interview 
techniques to assess the impact of those services on the users. 

4.3 Methodology: General approach 

Traditional survey methods, with predefined characteristics, and quantitative indicators, 
measures and/or scales, as discussed in Chapter 2, arc not suitable for exploring user 
motivation, and the associated bases for evaluation. Instead, the methods used for this task 
here are derived from: 

1 . Grounded theory approach , which suggests how to build models - in this case a 
taxonomy - from the bottom up, using raw data (Corbin & Strauss, 1990). 

2. Content analysis of texts , which suggests reduction to manageable size and 
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classification of large quantities of text - in this case using texts of interviews 
with users (Weber, 1990). 

3. Principles of taxonomies in social sciences, which suggests the construction and 
properties of such taxonomies. (Bailey, 1994). 

In testing the derived taxonomy we used standard statistical methods, to assess the 
degree of consistency in its application by two coders. More sophisticated tests, which lead 
to "label-specific" assessments of consistency have been developed as part of this work, and 
will be reported elsewhere. 

4.4 Methodology: Specific procedures 

The open-ended questions (see questionnaires in Appendix A) revolve around: 

1 . Reasons why a user came to a library or accessed a library service at this time. 

2. What did the user get out of the use; what benefits did he/she receive. 

3. What would a user do otherwise, if the service were not available at that 
library. 

4. Questions asking the user to elaborate on why he/she has given a particular 
score on Likert scales related to results of use 

The answers to the open-ended questions were tape recorded during the interview. 
(Detailed statistics on number of interviews at each library and service are given in Part 2 of 
this report dealing with the overall methodology). The tapes were transcribed, generating 
some 471 pages of text, and over 1.3MB of machine readable files. Since, in Wave 2, only 
those respondents who could be reached for callback interview were included, there is a total 
of 528 interview sets that have been analyzed in detail. 

Each user interview, containing all the questions and answers for a single user, was 
labeled with an interview number - this constituted the unit record for analysis. Unit records 
were arranged by wave, library, service and then by interview number. They were printed out 
and also provided on disks for further work by analysts. 

As described above, categories were not predefined, but were identified, defined, 
redefined, grouped, and tested from raw data itself, that is from interviews. This was an 
iterative process, as follows: 

1. One analyst went through all the interviews (unit records) and identified 
representative keywords and phrases as candidates for development of 
categories. He held frequent discussions with other team members about the 
selection of given candidates. The process analysis involved iteration, frequent 
revision and consensus. 

2. The analyst grouped the candidate keywords and phrases into categories 
associated with specific topics of questions and gave each category a label or 
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descriptive name, and a set of keywords to be used by other coders, in 
applying the labels. Generally the label was taken from the users' vocabulary. 

3. In this way a first cut was constructed. This is a working classification scheme 
(not yet a completed taxonomy) to be used for testing and later as a basis for 
derivation of revised versions. To facilitate analysis, codings were grouped 
into broad classes. 

4. The original analyst then went through the whole set of interviews (all unit 
records), and classified user responses according to the scheme. 

5. The analyst then wrote a set of instructions for the categorization of user 
responses, using the headings and categories in version 1 of the classification. 
These instructions were used to test whether the specific categories as derived, 
can be reliably applied in categorization of users' responses to questions in 
interviews. Appendix B contains version 1 of the scheme and the instructions 
for encoding of texts using that scheme. This constitutes the Empirical 
Taxonomy (ET). 

6. lasting followed. Two analysts, not engaged in the process previously, were 
given the interviews (raw data), version 1. of the classification scheme, and 
instructions for coding, and asked to repeat the process to confirm interjudge 
agreement. 

7. Then another analyst (member of the project team), worked with the broad 
classes and subclasses to arrive at generalizable conceptual groupings that 
unify the general conceptual scheme and the specific categories derived and 
tested in version 1. This is the second cut or version 2 of the scheme, which is 
a taxonomy. 

8. That analyst then went again through all the interviews (raw data) and tested 
the version 2 scheme, making adjustments in the entries as suggested by the 
data in view of the taxonomy and editorial changes as necessary. 

9. The result is version 3 of the scheme. Thus, version 3 is a result of both the 
previous two versions and repeated consultation and tests involving raw data. 

• Version 3 is the Derived Taxonomy (DT), which is presented in this Chapter. 

10. A translation or equivalency table was made to connect entries between 
versions 1 and 3. This translation table is also presented in Appendix B. 

Version 3 (DT) has been given to two more analysis, not previously involved with the 
project. They will rccode the raw data using that scheme. This test is in progress and will 
extend beyond the present project, and results will be reported elsewhere. 

To sum up, once the questions had been formulated, the scheme was developed from 
the bottom up. Users' answers to questions were the basis, and the only basis, for its 
development. While these methods resulted in a specific product, the methods themselves are 
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'generalizable and can be used in development and testing of" the taxonomy in all kinds of 
libraries and information centers. 

4.5 Structure and attributes of the taxonomy 

The taxonomy can be thought of as a faceted classification, incorporating levels as 
subdivisions. The structure of levels is as follows (where X. are letters and n. are numbers in 
the codes): 

X. General classes 

X.n Subclasses 

X.n.n Subclasses 

X.n.n.n Specific categories 

The classification is not of uniform depth, and grows according to the responses 
analyzed. 

Here are some of the attributes of the taxonomy as structured: 

1. Relations: The relations within and between different levels of the taxonomy 
(classes, subclasses, and categories) are based on set-member relationship, or 
detailed elaboration, as in faceted classifications. A class X has subclasses 
which are its members; subclasses are sets with other subclasses or categories 
as members. A specific category "Research" belongs, with a number of other 
categories, to a subclass "Task"; subclass "Task" belongs, with a number of 
other subclasses, to a class "Reasons." This is a polyhierarchical structure. 

In this sense, the relations are weaker than they would be in strictly ordered 
hierarchical schemes. The relation can also be thought of as a denotation-connotation 
relation. Specific categories denote an object and connote the class to which they belong. 

Selected user utterances are grouped into categories that denote (label) those 
utterances. In turn, the categories are grouped in subclasses that connote (imply) a common 
attribute shared by the set of categories, finally, subclasses are grouped in classes that 
connote an attribute shared by the set of subclasses. The principles for selection of given 
denotations and connotations, and their structure are based on the objectives, limitations and 
premises given in sections 4.1 and 4.2. 

2. Exclusivity: Broad classes (labeled by letters alone) are considered mutually 
exclusive. However, for the rest, the structure does NOT imply mutual 
exclusivity between categories or between subclasses. Categories may overlap 
and there is a certain degree of redundancy among a number of them. And, of 
course, an utterance may include several conceptually distinct contents. 

Thus the interpretation can also differ to a degree, and more than one category may be 
assigned to a given user utterance describing some reason, interaction or result. 
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'Correspondingly, several reasons, interactions or results may represent the same specific 
category or subclasses from the taxonomy. The relations can be one-to-many, and many-to- 
one. 

3. Uniqueness to individuals: Each user-service value relationship is unique. 
Consequently, value may be unique to an individual user. What is valuable to 
one may not be valuable to another, or the values may differ. What is valuable 
to the same user in one situation, context, or time slot, may not be in another. 
Value is relative. Thus, a search for classification of value, as done here, is an 
attempt to find recurring themes, with full realization that in practice there may 
be wide divergence among individuals. 

4. Open endedness: The taxonomy was derived from a sample of users. While the 
sample was large and representative of a number of library services, there 
could be other users and services which would call for additional specific 
categories of value. By necessity, any user-based taxonomy of values is not 
complete, but is open-ended. Additions can be anticipated and included. 

5. Stability: We hope, however, that the three general classes will remain stable; 
that subclasses will be relatively stable, and that the greatest need for change, 
as this taxonomy is adapted will be at the level of the "leaves" of the tree. At 
the level of specific categories level changes will almost certainly appear in 
further versions, particularly in relation to adjustments to fit specific 
applications in different libraries. 

4.6 Contents of the taxonomy 

Exhibit 4.1. lists the general classes and subclasses in the taxonomy, without the 
specific categories under each. This figure is intended to show at glance the overall structure 
of the taxonomy. The three general classes, A. Reasons, B. Interaction, and C. Results, 
represent the facets related to value in using a library service, as defined and discussed in 
section 4. 1 . 
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Exhibit 4.1. 
DERIVED TAXONOMY OF VALUE 
IN USING LIBRARY SERVICES 

General classes 

A. REASONS for using a library service 

A.1 For a TASK 
A.2 For PERSONAL reasons 
A.2.1 Cognitive 

2.2 Emotive 

2.3 Substitute choice 

A. 3 To get an OBJECT or perform an ACTIVITY 

A. 3.1 Physical (tangible) objects 

3.2 Intangible objects 

3.3 Perform an activity or work 

B. INTERACTION with a library service 

B. l RESOURCES - availability, accessibility 
B.2 USE of resources, services 

B.3 OPERATIONS and ENVIRONMENT 

B. 3.1 Policies, procedures 
B.3. 2 Facilities, organization 
B.3. 3 Staff performance 

B.3. 4 Equipment performance 

B. 4 Possible SUBSTITUTIONS for given interactions and library 

service(s) that were used 

C. RESULTS of using a library services 

C. l COGNITIVE results 
C.2 EMOTIVE results 

C.3 ACCOMPLISHMENTS in relation to tasks 
C.4 Meeting EXPECTATIONS 
C.5 TIME aspects 
C.6 MONEY estimates 



Exhibit 4.2. presents the whole taxonomy. We find three general classes, 23 
subclasses, and 101 specific categories. This is our third version, which we call the "Derived 
Taxonomy" (D1). 

Description and definition of the contents for each class and subclass are given in 
sections 4.6.1 to 4.6.3. Section 4.7 provides a discussion of encoding and overlaps. 

4.6.1 Class A. Reasons 

Class "Reasons" covers the causes, motives, bases, purposes, and/or rationale 
underlying the use of library services. Why do uscr.s use a library service? What do users 
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want to get from a library service? We have subdivided Reasons into three subclasses: 



A.l Task: Covered are the activities, work, or problems with which the user is 
engaged that were the direct cause for using the library service. What are the users 
doing that prompted the use of the service? What were the users working on or 
wanting to work on? What problems were the users faced with that brought them to 
the service? Included are tasks such as: research, bibliography compiling, class 
assignment, project proposal, and a host of others. Most of the times there is some 
tangible element in Tasks, an element that can be directly observed. 

A.2 Personal reasons: Covered are the private, individual reasons for using the library 
services. Most of the time they are intangible. What is the motivation of the users for 
using a library service, as far as knowledge or emotion is concerned? The Personal 
Reasons are further subdivided into three subclasses: 

A.2. 1 Cognitive , related to various aspects of learning and knowledge, staying 
current, or orienting oneself; 

A.2.2 Affective (or emotive) , related to a person's own emotional feelings or 
desires for using the services, like relaxation, stress reduction, feeling of 
satisfaction; 

A.2. 3 Substitute choice, related to a personal decision to use the given library 
service rather than some other choice. 

A. 3 To get an object or perform an activity: Covered are reasons associated with what 
the users directly intend to request from a service or what activities they intend to 
engage in at the library. What do the users actually want to get from the service? 
What will they do in the library? This subclass is further subdivided into: 

A.3.1 Physical (tangible) objects: Getting a book, or other item available from 
a library service; 

A.3.2 Intangible objects: Getting information, facts, data; being directed to 
other sources of information; 

A.3.3 Perform an activity or work: Studying, searching, browsing and other 
activities a user intends to perform. 
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Exhibit 4.2. 



Derived TAXONOMY OF VALUE 
IN USING LIBRARY SERVICES 



General classes and specific categories 



A. REASONS for using a library service 



A. 1 For a TASK 



A. 1 . 1 Research 

1 .2 Dissertation/thesis 

1.3 Project work 

1.4 Professional and other work 

1.5 Paper, report - writing, starting 

1.6 Book - writing, starting 

1.7 Bibliography, references - compiling, checking 

1.8 Class assignment 

1 .9 Exam 

1.10 Teaching - preparation, gathering materials 

1.11 Presentation 

1.12 Grant, project proposal 

1.13 Job search, application, interview 

1.14 Review, assessment - of a book, proposal, application 

1.15 Planning for some activity, work 

1.16 Delegated work - doing it for somebody else (professor, colleague, 



friend...) 
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A. 2 For PERSONAL reasons 



A. 2.1 Cognitive 
A.2.1.1 
2.1.2 



2.1.3 



Learning something, confirming something 
Staying current, catching up with an 
area, topic 

Orienting oneself to the library, resources, services, 
equipment; learning how to use them 
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Derived TAXONOMY OF VALUE 
IN USING LIBRARY SERVICES 

Exhibit 4.2 (cont'd.) 



A.2.2 Emotive 




A.2.2.1 


Relaxing, pleasure, recreation, leisure 


A.2.2.2 


Reducing stress, worry 


A.2.2.3 


Feeling satisfaction, accomplishment 


A.2.3 Substitute choice 


A.2.3.1 


Using this library service instead of other choices - 




other information resources, services people 


A.3 To get an OBJECT or perform an ACTIVITY 


A.3.1 Physical (tangible) objects 


A.3. 1.1 


Getting a book, periodical, article, recording, image, 




film... 


A.3.1. 2 


Using interlibrary loan to get an item 


A.3. 1.3 


Using recall to get an item 


A. 3. 2 Intangible objects 


A.3.2.1 


Obtaining information, facts, data; clarify something 


A.3.2.2 


Pointing to another source(s) of information within or 




outside the library 


A.3.3 Perform an activity or work 


A.3.3.1 


Studying in the library 


A.3.3.2 


Viewing a film, listening to a recording 


A.3.3.3 


Searching electronic resources - catalogs, databases... 


A.3.3.4 


Searching print resources - catalogs, indexes, tables of 




contents... 


A.3.3.5 


Browsing 


A.3.3.5 


Copying 


A.3.3.6 


Use computers for non-library task (where available 




as a service) 


A.3.3.7 


Performing other non-library activities, work... 



47 



APLAB-B%D\CLR\FINAL.REV\FINALBIG.51 Copyright (c) 1995 



March 21, 1995 P. 45 



Derived TAXONOMY OF VALUE 
IN USING LIBRARY SERVICES 

Exhibit 4.2 (cont'd.) 

B. INTERACTION with a library service 

B.l RESOURCES^ SERVICES - availability, accessibility 
B.l.l Availability of desired item(s) 

1.2 Completeness of given resource, service - degree of 

1.3 Currency, timeliness - degree of 

1 .4 Accessibility to a given resource, service - degree of 

1.5 Hassle, frustration in accessing the resource, service or in getting it - 
degree of 

B.2 USE of resources, services 

B.2.1 Convenience in using the resource or service - degree of 

2.2 Effort required in using it; ease of use - degree of 

2.3 Frustration in using it - degree of 

2.4 Effort, ease of getting from one resource or service to a 
complementary or subsequent one - degrpe of 



B.3 OPERATIONS AND ENVIRONMENT 

B.3.1 Policies, procedures 

B.3. 1.1 Clear - degree of 

3.1.2 Conducive for ease, convenience, effectiveness of 
access, use - degree of 

3.1.3 Requirements upon users, fairness, reasonableness - 
degree of 

B.3.2 Facilities, organization 

B.3. 2.1 Space, - degree of adequacy 

3.2.2 Physical layout and organization - degree of quality 

3.2.3 Adequacy, quality of intellectual organization of 
resources, materials, services - degree of 

3.2.4 Comfort of facilities - degree of quality 
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Derived TAXONOMY OF VALUE 
IN USING LIBRARY SERVICES 

Exhibit 4.2 (cont'd.) 



B.3.3 Staff performance 



B.3.3.1 Knowledgeability, expertise - degree of 

3.3.2 Helpfulness, empathy - degree of 

3.3.3 Efficiency - degree of 
B.3.4 Equipment performance 

B. 3.4.1 Technical functioning - degree of quality 

3.4.2 Availability and clearness of instructions, guides 

3.4.3 User friendliness - degree of 

3.4.4 Difficulty in operations - degree of 



B.4 Possible SUBSTITUTIONS for given interactions and library service(s) 
that were used 

B.4.1 Use another service in the library 

4.2 Go to another library 

4.3 Use other facilities, services, resources outside of libraries; get the 
item through a department, company, society 

4.4 Purchase the book, publication, recording... 

4.5 Subscribe to a service, journal 

4.6 Consult with a person - friend, professor, colleague... 

4.7 Borrow the desired book, or other item from a person who has it 

4.8 Use interlibrary loan 

4.9 Do more work on one's own 

4.10 Do nothing 
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Derived TAXONOMY OF VALUE 
IN USING LIBRARY SERVICES 

Exhibit 4.2 (cont'd.) 



C. RESULTS of using a library services 



C.l COGNITIVE results 

C.l.l Learning something, stretching knowledge 

1.2 Reinforcing knowledge, reconceptualizing 

1.3 Changing viewpoint, outlook, perspective 

1.4 Getting ideas, perspective, conceptualization how to proceed 

1.5 Serendipity - getting ideas about different, tangential things 

1 .6 Getting no new ideas; did not learn anything 

C.2 EMOTIVE results 

C.2.1 Sense of accomplishment - increase, decrease 

2.2 Sense of failure - increase, decrease 

2.3 Gaining confidence - degree of 

2.4 Comfort, good feeling, happiness - increase, decrease 

2.5 Frustration, stress - increase, decrease 

C.3 ACCOMPLISHMENTS in relation to task(s) 

3.1 Contribution to accomplishing or proceeding with task at hand; 
facilitation of or help with work - degree t >f 

3.2 Fulfilling assignment - degree of 

3.3 Providing access to people or other sources of information 

3.4 Providing for a next step, next task, next information seeking 
activity - degree of 



C.4 Meeting EXPECTATIONS 

C.4.1 Getting, obtaining what needed, expected or requested - degree of 

4.2 Getting too much 

4.3 Getting nothing 

4.4 Uncertainty about or confidence in what gotten - degree of 

4.5 Exceeding expectations, getting additions to what expected - degree 
of 

4.6 If not gotten what expected, degree of hurt; seeking of substitute 
actions 
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Derived TAXONOMY OF VALUE 
IN USING LIBRARY SERVICES 

Exhibit 4.2 (cont'd.) 



C.5 TIME aspects 


C.5.1 


Saving time as a result of using the service - amount, comparison 


5.2 


Wasting, losing time in using the service - amount, comparison 


5.3 


Waiting time to access the service - amount 


5.4 


Quickness, speed of service 


5.5 


Time it takes to figure out or use the service - amount 


5.6 


Time available or allocated for use of the service - sufficiency 


C.6 MONEY 


estimates 


c & 1 


JL-ollHlo.LC' Ul a. \l\JllCLL VdlUC VJ1 l&oUXLo \JUl<XLll<>\A 11 VJIii a. gl V t>ll oCl Vlvt-, 




or of information obtained 


6.2 


Estimate of a dollar value saved because of use of the service 


6.3 


Estimate of a dollar value spent in using the service, or the actual 




amount spent 


6.4 


Estimate of what may be spent on a substitute service or activities 




for similar results 


6.5 


Estimate of dollar value lost in cases where service was not 




available in the library or the use was not successful 
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4.6.2 Class B. Interaction 



Class "Interaction" covers the assessment by users of qualities of various aspects of 
library services. How do the users assess or evaluate the encounters with the library in 
seeking and using a service? What else would they have done, if the service were not 
available? We have divided Interaction into four subclasses: 

B.l Resources, services: Covered are availability and accessibility of given items and 
services. Availability asks: Does the library have a given resource, item or service? 
Does it have a given book or journal and is it available for use on this occasion? 
Also included are: How current or timely is a given resource or service? 

B.2 Use: Covered are aspects connected with use of a resource or service. 
Accessibility asks: what are the barriers to using it, in cost, time or inconvenience. 
How convenient is it to use the service? What effort is required or how easy is it to 
use? What is the degree of frustration? How much effort is required to get from one 
service to another (e.g. from searching for and finding references to getting articles)? 

B.3 Operations and environment: Incorporates aspects related to the working and 
environment of the service vis-a-vis the user. Four subclasses are included: 

B.3.1 Policies, procedures: How clear are they? How conducive are they for 
access and use? How reasonable and fair are the requirements? 

B.3. 2 Facilities, organization: What is the adequacy, quality of service's space, 
layout and facilities, and of intellectual organization of materials? 

B.3. 3 Staff performance: What is the knowledgeability of staff in the service? 
How helpful and efficient are they? 

B.3. 4 Equipment performance: How well does equipment function? Are 
instructions available and clear? How difficult it is to use? How user friendly is 
it? 

B. 4 Substitutions: Covered are indications by users of possible other services and 
lines of action, if the given service would have not been available. What would the 
users do to get a similar service or desired results? Go to other libraries, or 
information services? Consult people? Put efforts elsewhere? Do nothing? 

4.6.3 Class C. Results 

Class "Results" covers the outcomes of interactions with library services. Wlvit did the 
users get out of the service? What did they accomplish? Were their expectations met? How is 
the service related to time and money? We have divided Results in six subclasses: 

C. l Cognitive results: Relate to the state of knowledge of users: Did they learn 
something? Reinforce their knowledge? Change perspective? Get ideas on how to 
proceed? Any serendipity? Got no new ideas, learned nothing? 
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C.2 Affective or Emotive results: Relate to the state of users' emotions and feelings. 
Did they get a sense of accomplishment? Did they gain confidence? Were they feeling 
happy, frustrated? 



C.3 Accomplishments: Relate to task at hand. What was the contribution to the tusk? 
Did it help the assignment? Was access to people or other information sources 
enabled? Did it help to continue seeking further information? 

C.4 Expectations: Relate to meeting of expectations. Got what needed? Too much? 
Too little? Nothing? How much confidence in what gotten? If not gotten what 
expected, how much hurt? 

C.5 Time: Relates to a number of aspects of time. How much time may have been 
saved as the result of service? How much time may have been wasted? What was the 
waiting time to get the service? Time needed to use or figure out the service? Was the 
allocated time sufficient? 

C.6 Money: Relates to estimates of dollar values. What is the estimate of dollar value 
received from service? Dollars possibly saved? Dollars actually spent? How much 
might have been spent on a substitute service? How many dollars may have been lost, 
when the service was not available or not successful? 

4.7 Encoding and overlap 

In this section we elaborate on the rationale for instituting a scheme of classes, 
subclasses and categories, and the relations between them. This is of particular importance for 
encoding of users' responses and fitting them into given categories. Thus, this section can be 
considered also as a general part of coding instructions. 

As mentioned, user answers can receive several codes, and conversely, the same code 
can be assigned to several answers. 

Specific categories in subclass Tasks are not exclusive of tho^ in Personal Reasons. 
Most of the time a distinction can be made by thinking of Tasks as being tangible and also 
represented in the outer world, while Personal Reasons are intangible and represented in the 
inner world of people, but categories from both may be applicable when users explain why 
they used a service. Similarly, categories in subclasses under Interaction (Resources, Use and 
Operations), may be simultaneously applied, as may those under Results. Furthermore, more 
than one category under Tasks (or any other subclass) may be used to classify user answers. 

In probing for answers as to why the users use the library we found that the users 
often restrict themselves to the objects they want to get ("To get a book"), or an activity to 
perform ("Do a search"); they do not think to answer with the underlying reasons which 
drove them to get an object or do an activity, like "class assignment" or "paper writing." One 
has to probe to get to these underlying reasons - the userc t'link of all of them as one, even 
though they evidently exist in layers. Similar examples can be given for a number of other 
questions. Users tend to bundle reasons, as they do interactions and results. This is the main 
reason for a lack of exclusiveness. As a practical matter, we substantially revised the 
question order between Wave 1 and Wave 1.2 in order to reduce the tendency for users to 
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'think in this way. 

On the other side of the coin, the action verb "to get", and its synonyms span classes 
of the DT. Categories related to "get" appear in two classes, Reasons and Results, but with 
significant distinction. Under Reasons, and subclass To Get an Object or Perform an Activity, 
there is an intention: "Why did you use the library today?" "To get a book." Under Results, 
and Accomplishments, there is an outcome: "What did you get out of using the library 
today?" "I got a book." Although in this case the answers to both questions are similar, they 
are distinct: the first refers to a reason for using the service and an intended action, while the 
second one refers to a result and an accomplished action. Often, they follow each other, but 
at times "To get a book" is not followed by "I got a book." The answers are not symmetrical. 
We found that Reasons and Results are at times difficult to distinguish while encoding, thus, 
in instructions for encoding (Appendix B) special attention was given to describing these 
distinctions in detail. 

4.8 Statistics and testing 

Analysis of the Coding of the Open-ended Responses under the Empirical Taxonomy. 

Analysis of the open-ended responses was an iterative process which involved a great 
deal of informal adjustment and communication among the coders. Lacking comprehensive 
coding support software, we can best summarize the process through a qualitative discussion 
of the coding responses shown in the following tables. In most of these tables we show only 
those responses that occurred in more than 10% of the cases. 

Each response text was coded according to one of six coding category schemes that 
had been developed. Some of the earlier schemes were later dropped and were not used in 
coding data gathered in the second wave of the survey. We allowed up to three codes to be 
assigned to a particular response text. Therefore, the codes are analyzed using a multiple 
response capability supported by the SPSS program. We summarize the results of that 
analysis here. Full detailed tables of results are given in Appendix C. Note that while some 
schemes were applied to questions in every instrument, others had more limited application, 
resulting in a smaller number of valid cases. 

In the table below, the column headed "Code" contains the numerical code used to 
represent the classification in the Empirical Taxnomy. The translation table is given in 
Appendix B. The total number of questionnaires on which such a response was coded is given 
in the column headed "Count". This number is shown as a fraction of all the responses coded 
in the next column. It is shown as a fraction of all the questionnaires in the last column. 
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'Since the same code may be assigned more than once to one questionnaire, the last column 
sums to more than 100%. 



Group $CHOOSE Coded on Choose ("Why- 
library? " ) 

Category label 

Tota 

Got everything I wanted 
Save time 
Helpful finding 
Could have been more 
Got most 

More easy, convenient 

452 missing cases; 76 valid cases 



did you choose that grade for the 







Pet of 


Pet of 


Code 


Count 


Responses 


Cases 


responses 


109 


100 . 0 


143 .4 


51 


22 


20.2 


28.9 


21 


14 


12. 8 


18.4 


12 


13 


11. 9 


17. 1 


53 


11 


10 . 1 


14.5 


52 


10 


9 . 2 


13. 2 


24 


10 


9.2 


13 .2 



For example, for the category called "Choose", coded according to the detailed scheme 
given in Appendix B, there are an average of 1.43 codes assigned per case. The actual 
number of responses given is shown in the column headed "count", and the table has been 
sorted in decreasing order of that count. However, we will focus our attention on the last 
column of the table which shows the percentage of all cases coded in which a particular 
response was given. For example, the response "got everything I wanted" occurred in 22 
cases, out of total of 76, representing 20.2% of all the (109) responses, but representing 
28.9% of all the cases coded. In other words, nearly 1/3 of the responses coded according to 
this scheme included the concept that the patron got everything that he or she wanted. 

The uex*. ranking score has to do with saving time. This occurred in 18.4% of the 
cases. Ranked very close to it is that the library and its staff were helpful in finding the 
needed information. The fourth ranked score is one which generally appeared in explanation 
of a negative or low score assigned to the library, and it is that the library or the service 
could have been more effective. Nearly tied with this are two responses which address the 
acquisition of information. The first, at 13.2% is an indication that the patron got most of 
what was needed, while the second, equally frequent, was an indication that the process was 
somehow easy or convenient. 

No other response (details are given in appendix C) scored above 7% of the cases. 

The table below shows 242 cases that were scored according to the code group 
describing what the patron would have done if the service had not been available at the 
present library, or what the patron is going to do because the service has not been available. 
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•Group $DONE Coded What Would Have Done (".. if you had not gotten what you 
needed? " ) 



Category label 







Fee or 


fee or 


Code 


Count 


Responses 


Ca.s gs 


ponses 


jl/ 


inn n 
X u u . u 


1 1 1 n 


4 


44 


13 . 9 


18.2 


42 


38 


12 . 0 


15.7 


5 


25 


7 . 9 


10.3 


41 


21 


6 . 6 


8.7 


31 


19 


6.0 


7 . 9 


26 


19 


6 . 0 


7 . 9 


511 


18 


5.7 


7 . 4 



Another library 
Another university 
Another place 
A public library 
Interlibrary loan or recall 
Card catalog 
Buy items 

286 missing cases; 242 valid cases 

In this case we extend the response threshold down to 7% of the responses in order to 
cover some that are of importance to library practice although they did not occur as 
frequently as we might have expected. The first four categories (.covering 18.2, 15.7, 10.3, 
and 8.7% of the responses) indicate looking in another place, which is likely to be library, 
either named as a library, described as being a university, or simply described as being 
another place. Note that these scores can not be summed because of the multiple response 
feature. In other words, we don't know the total number of cases in which one of these four 
responses appeared. Only after these four leading activities do we come to the 7.9% of the 
cases in which the patron is going to make use of usual library services. The frequency is the 
same for use of interlibrary loan or recall and for further use of the card catalog. Next after 
this, occurring in slightly more than 7% of the cases is an indication that the patron would 
have to buy the book. 

It appears then that either because of press of time or for lack of promotion of the 
service, two of the library's most important tools for serving patrons who are frustrated in 
their original search are no more prominent in the patron's thinking and planning than is the 
presumably dire alternative of buying an item. 

There were 185 cases coded by a set of concepts describing how the patron's goal or 
task or project would have been hurt by failure to obtain the information. See the table 
below. 



Group $HURT Coded by how it would hurt (" 
what you sought?") 



your project if you did not gee 



Category label 



Code 
Total responses 



Not able to complete 

Taken more time 

Gotten less information 

Other information source 

343 missing cases; 185 valid cases 



10 
50 
31 
34 



Count 



Pet of 
Responses 



Pet of 
Cases 



230 


100 


0 


124 


3 


66 


28 


7 


35 


7 


53 


23 


0 


28 


6 


26 


11 


3 


14 


1 


20 


8 


.7 


10 


8 




Heading the list is the statement that it would have been impossible to complete the 
project, which was reported by more than 1/3 of the patrons. This is a clear indication of the 
application aspect of the patron's conceptualization of the library service. The second 
category "taken more time" occurs in more than 1/4 of the cases and clearly refers to the 
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acquisition process. The third and fourth categories ("gotten less information" and "other 
information source") clearly expose the conceptual aspects of the patron's use of the library. 

Thus, the four leading concepts appearing in the patrons' explanations of how they 
would have been hurt correspond more clearly than the codings given before to our broad 
conceptual scheme of acquisition, cognition, and application. 

There were 174 cases coded according to the reason that seme particular score had 
been assigned. The top four categories express aspects of the patron's conception of the 
reliability and the value of the service. Note that it is not apparent from an examination of 
the questions that this dimension of the perception of service would have been probed. 
However, we find, in 17.8% of the cases, confidence in the service, and in 16.7% of the cases 
(overlap not determined) the statement that the patron had used the service in the past. Then, 
presumably accounting for negative or low scores, are the concepts of "incomplete", "not 
comprehensive", or "not useful". Finally, the category of generally "helpful" occurs in just 
below 10% of the responses. 



Group $REASON Coded according to reason ( " . . . for 


giving 


that score 


• ) 






Pet of 


Pet of 


Category label Code 


Count 


Responses 


Cases 


Total responses 


220 


100.0 


12 6.4 


Confident in service 42 


31 


14.1 


17.8 


Used the service in the past 43 


29 


13.2 


16.7 


Not complete , comprehens ive, useful 40 


22 


10.0 


12 . 6 


Helpful 32 


17 


7.7 


9 . 8 



354 missing cases; 174 valid cases 



There were 182 case of responses coded according to the way in which the library 
service helped the patron's task or project. These, as might be expected, cover a variety of 
aspects of the cognitive dimension. For example, the leading category is that the service 
provided information or references. This is, of course, also related to the acquisition 
component, but the appearance of the word information suggests that the patron is already 
directed towards the cognitive aspects of the interaction. "Helped me start", occurring in 
14.8% of the cases, has to do with the application process, so also does "complete or help my 
work". The fourth listed category, occurring in 12.1% of the cases was very interesting 
because it is clearly cognitive and represents the notion that the service provides "ideas, 
insight, or perspective". 



Group $WAY Coded according to way (".. that the library helped you".) 







Pet of 


Pet of 


Category label 


Code Count 


Responses 


Cases 




Total responses 203 


100. 0 


111.5 


Information/ references 


24 58 


28.6 


31.9 


Helped me start 


14 27 


13.3 


14.8 


Complete or help my work 


23 23 


11.3 


12 . 6 


Ideas, insight, perspective 


22 22 


10.8 


12 . 1 



346 missing cases; 182 valid cases 



We turn finally to the most extensively applied group of coding responses, which were 
coded according to what the patron "got out" of the service. These codes were also applied 
to questions having to do with the value or benefit of the service. In this case, it is very 
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gratifying to see that the three leading categories correspond to our three broad conceptual 
categories. Getting a physical object was cited in 86.9% of the cases. Obtaining information 
or knowledge was cited in 77.1% of the cases, and "for a paper" was cited in 72.5% of the 
cases. 

Looking at the top three reasons reported we see here clearly the extreme importance 
of the acquisition, cognition, and application aspects. That is "get physical object" is 
acquisition, "get information or knowledge" is clearly cognition, and "for a paper" is clearly 
application. Interestingly, however, application may play a much higher role in people's 
thinking because the next several categories, "work", "orientation", "task completion", "for 
research", and "for a class" all clearly relate to the application aspect. At 31.8% we find 
"saving time" and just below it "convenience and efficiency", which probably refer to 
qualitative characteristics of the acquisition process. The next two categories were used to 
code negative experiences, neither occurring very frequently. However, 20% of the responses 
included some statement that information or resources were not available, and 22.5% 
indicated that in some way the service did not fulfill the immediate goals. 



Group SGETOUT Coded by What 


you Getout ( " . . . of 


using the 


service . " ) 










Pet 


of 


Pet of 


Category label 


Code 


Count 


Responses 


Cases 




Total responses 


3497 


100 


0 


662 .3 


Get physical Object 


31 


459 


13 


1 


86.9 


Information, knowledge 


32 


407 


11 


6 


77 . 1 


For a paper 


52 


383 


11 


0 


72 .5 


Work 


2 


271 


7 


7 


51 .3 


Orient 


1 


265 


7 


6 


50.2 


Task completed 


36 


212 


6 


1 


40 .2 


For research 


51 


192 


5 


5 


36.4 


For a class 


■56 


174 


5 


0 


33 . 0 


Save time 


33 


168 


4 


8 


31.8 


Convenience, efficiency 


34 


146 


4 


2 


27 .7 


Didn't fulfill goal 


44 


119 


3 


4 


22 .5 


Not available 


41 


110 


3 


1 


20.8 


Positive (People were nice) 


941 


53 


1 


5 


10 . 0 



0 missing cases; 528 valid cases 

Last in the group of responses occurring more than 10% of the time is the simple 
statement that the experience wa f positive because the people at the library were nice. As an 
aside we note that this suggests that alarming indications in literature that simply behaving 
nicely will completely distort patron's perception of the quality of service are perhaps 
unfounded. Only 10% of the cases cite this kind of reason, after an overwhelmingly more 
frequent appearance of reasons having to do with acquisition, cognition, and application. 

4.9 Theoretical derivations 

While theory is not a primary focus of this work, our observations lead to some 
theoretical implications. We believe that the general structure of the taxonomy (Exhibit 4.1) 
may also serve as a model of values in use of library and information services in general. 
That is, these are the classes of variables that enter into assessments of value. We now 
comment about three elements that may have to be incorporated in a general theory of valuc- 
in-use, or any more specific theory of values in using library and information services. 
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The first element relates to layers or levels. It seems that users consider values in a 
progression from a surface level that deals with immediate objects and activities, to deeper 
levels that deal with motivations - accomplishment of tasks, emotive reasons and the like. 
At times, the different levels are intertwined, but at other times one level may be singled out 
to dominate in an assessment of value. 

The second element relates to stages in the process whose value is assessed. It seems 
that assessment of value by users proceeds through several stages. We can distinguish at least 
three stages, and at each stage a different set of values may be applied. To illustrate: "Getting 
a book" is an immediate resuit, which may after a time become "learned something" - a time 
during which a user read and absorbed, understood and/or assimilated what is in the book. In 
cases where users have to perform a task or resolve a problem ("complete a report") there is a 
further, stage, also reached after a passage of time . "Getting a book" may be characterized as 
the first or acquisition stage; "learning something" as the second or cognitive stage; and 
"compL ting a report" as the third or application stage. 

At different stages, users may refer to a different scale of valuation. Thus, different 
expressions of value may be separated by time periods or stages. While our taxonomy did 
not follow such a model, the stage value assessments are still present. We have incorporated 
the value assessments of the first or acquisition stage under categories associated with 
Interaction, and the second (cognitive) and third (application) stage under Results. 

The third element relates to exchanges. Value, particularly in economics, is often 
represented in terms of exchange of an object or service for and some tangible economic 
medium of exchange such as money or time. As discussed earlier, time, rather than money, 
was found to be the most effective dimension for assessing value. As have others, we found 
that users have great difficulty in assigning monetary value to library and information 
services. But they refer easily and often to time ("saved me time", "loss of time"...). 

Various aspects of time may be the strongest indicator of exchange value for library 
and information services, and this should be reflected in associated models and theories. In 
our taxonomy, we have chosen Time as a major characteristic of value, standing as a 
subcategory of its own. We also included Money, because some users refer to money when 
talking about the value of service. 



RJC 
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5 COMBINING COST AND VALUE DATA 



5.1 Impact Data 

Given the negative results of our efforts at scale development, we have explored the 
potential of combining two measures in a non-standard way. In particular, we have 
information on the assigned dollar value of the service, on the length of time spent using the 
service, and on the question "was the value worth the time?". We first explored the dollar 
value by itself. The distribution of this variable shows peaks at convenient numbers, and is 
highly skewed, with a concentration at very low values, and a high mean, influenced by a few 
outliers. We trimmed the variable by assuming that estimates in the range of $4,000 to 
$5,000 could be economically realistic, but that estimates of $50,000 could not be. We then 
explored the variation of the mean of the dollar value with the type of service. No 
statistically significant variation was found. 

As mentioned above, none of the scale variables yielded significant dependence on the 
service type. Similarly, analysis of the time spent yielded no significant dependence. 
We transformed the time spent to a logarithmic scale (based on natural logarithms), using a 
variable LogTime. This also did not show any significant variation with the service. 
However, the time spent by users at a library does represent some kind of an economic 
decision, and we sought to weight this new variable using the information about the relation 
between the benefit and the time. 

To do this we defined a composite variable: 

WtLogTime = LogTime * (B-l)/6. 

Here B is the scale value reported for "is the benefit worth the time". The highest value of B 
is 7, in which case the new weighted variable has the same value as the logarithm of the time 
spent. But for the lowest value of B (B=l), this weighted variable vanishes. Since the Time 
spent using the service is always greater than 1 minute, this new variable is thus either 
positive or zero. 

Analysis of variance revealed that this new variable is just barely able to resolve 
differences among the services. Specifically, the data let us reject the hypothesis that it has 
the same mean value for all the services. But further analysis (called "post-hoc" analysis) 
reveals that only the highest of the values found for the mean of the weighted logarithm of 
the time is significantly different from some of the other values. And, at that, it is 
significantly different only from the four lowest values. 

Thus the measure WtLogTime which is used in the remainder of this discussion is 
only a very weak discriminator of the value of the services. Nonetheless, it represents the 
best that we have been able to extract from the available data, and will serve to illustrate the 
concepts of Data Envelopment Analysis. 
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5.2 Combining Cost and Impact Data 



To illustrate how one can address both cost and value simultaneously we assemble 
here the data on the weighted log-time value measure, and the cost data, drawn from Chapter 
2. In order to produce a usable graphical representation we also represent the cost on a 
logarithmic scale, by calculating LogCost. 



Table 5.1. Weighted Logarithm of the Time Spent Using the Service combined 
with cost per unit of service. The services are defined in more detail in 
Table 1.2. (D-Materials Delivery. C=Catalog or online service. R=Reference 
Service. V=Reserve Collection. L=Entire Collection. X=Other service.) 



Code 


Mean 


StdDev 


N 


Cost 


LogCost 


CI 


1.4089 


1 


1017 


24 


0 . 


70 


-0 . 34 


C2 


1 . 8783 


1 


0654 


30 


0 . 


70 


-0.34 


Rl 


1.4113 


0 


7576 


47 


1. 


16 


0.15 


Dl 


1 . 1145 


0 


8669 


52 


1 


88 


0 . 63 


R2 


1.8642 


0 


9791 


47 


2 


00 


0 . 69 


VI 


1.3795 


1 


3662 


26 


2 


65 


0 . 98 


V2 


2 . 1946 


1 


5481 


7 


2 


65 


0.98 


LI 


1.5764 


1 


. 1947 


49 


6 


12 


1 . 81 


R3 


1 .289 


1 


.3758 


48 


6 


87 


1 . 93 


R4 


1 . 0997 


0 


. 6624 


23 


8 


18 


2 . 10 


V3 


1.3712 


1 


. 0751 


7 


8 


59 


2 . 15 


V4 


1 . 1797 


0 


.8765 


28 


8 


59 


2 . 15 


XI 


1.3037 


0 


.9727 


14 


2 


30 


2 . 51 


L2 


1.454 


1 


.3965 


17 


4 


62 


2. 68 


R5 


0 . 8463 


0 


. 5615 


10 


18 


.35 


2 . 91 


R6 


1.4841 




1. 094 


41 


20 


35 


2 . 93 


R7 


0 . 9672 


1 


.3678 


2 


18 


. 80 


2 . 93 


R8 


1 .784 


1 


. 6408 


7 


33 


.36 


3 . 51 


R9 


1.8322 


1 


.1015 


25 


35 


. 52 


3 . 57 


D2 


1.4433 


1 


.0171 


10 


36 


. 13 


3 . 59 



One of the services could not be independently costed, and was assigned the cost of 
"0", which cannot be transformed onto a logarithmic scale. Note also that when the unit of 
cost is taken to be the dollar, values less than $1.00 are represented by negative values of the 
logarithm of the cost. This does not represent a negative cost, but is just a reflection of the 
arbitrary unit of currency. 

5.3 The Principles of Data Envelopment Analysis (DEA) 

In data envelopment analysis (Ahn et al, Banker et al, Charnes et al, Seiford) service 
units, or functions, are represented by points on a graph such as the one shown in Figure 5.1. 
One axis represents the cost of the service, and the other represents the impact. Such an 
array of data would conventionally be analyzed by some type of regression analysis. What 
that does is produce the "average line". For data such as shown here the average line does 
not tell us very much, as the individual services are widely scattered. 

Under data envelopment analysis wc concentrate on the boundary defined by these 
points. In this case it is the line determined by the two left-most points, labelled CI and C2, 
and the highest point, labelled V2. These represent, among them, the three services which 
cannot be beaten. All other services have either higher unit costs, or lower impact (as 
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Figure 1. A Joint Plot of the Presumed Measure of Impact and the Determined Cost of a Unit 
of Service. Points closer to the upper left hand corner represent greater impact for the money 
spent on the service. 



measured by our surrogate) or both. In a sense this is a two-dimensional variant of the 
familiar notion of "benchmarking" which seeks to identify best practice for any particular 
service or function. Since different services are being considered together, the benchmark 
cannot be reduced to a single number, but must consider the cost and the impact separately. 
If we were confident in the validity of our measure we could in fact proceed to calculate the 
degree to which each other service is less than optimal. In the present, exploratory situation, 
such a display of analytical prowess would be unjustified. 

We note a few interesting features of the DEA plot. First, the catalog services are 
clearly least expensive. This is expected, particularly as we have not included the costs of 
developing the database to which it provides access. The catalog appear twice with the 
higher impact value (C2) being reported by users who were interviewed by phone. Thus it 
has a greater positive impact for those who use it from remote sites than for those who use it 
at the library (CI). This suggests that either it is used longer, or that the user adjusts the 
estimate of benefit to compensate for the effort required to go to the library. The reserve 
room services "V" have costs in the middle range, and impacts ranging from the highest to 
among the lowest. In fact, VI and V2 correspond respectively to student and faculty 
estimates for the same service. So do V3 and V4. Thus, in each case, the impact measure 
assigned by faculty exceeds that assigned by students. This suggests that the benefit, to the 
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■ faculty, of the reserve services is greater than the benefit to the students on a single visit. We 
must also note that faculty members typically have a single long interaction with the reserve 
room, and we have not separately costed the fulfillment of that interaction. Of course, this 
must be balanced against the fact that students probably use the reserve collection several 
times during a semester. 

The document delivery services "D" are not accorded a high impact and show 
substantial cost variation. However, this low impact is a reminder of the fact that we have 
not isolated impact with the clarity that we would like. Patrons were asked "how long they 
spent using the service". Thus, while it may take only a few minutes to request and receive a 
document, it may take hours to peruse that document and put it to some use. We return to 
this issue in Chapter 6. 

In the middle of the pack we find two situations in which a collection of some type 
was assessed as a whole: "L". Finally, there were several reference services at several of the 
libraries, and these, represented by "R" are scattered over the whole range of the DEA plot. 
Thus there is clearly a great deal more to be learned about how to measure the impact, and 
about whether (and if so, why) the cost, of achieving an impact varies so widely from one 
service to another. 
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6 DIRECTIONS FOR FURTHER STUDY AND 

APPLICATION. 



6.1 Development of Scales and Measures of Value 

6.1.1 Applyins the New Taxonomy 

Developing the codes for the open-ended responses was a complex iterative process. 
For example, we discovered that when respondents are discussing a number of different 
issues, they return to the same conceptual aspect of their use of the library. Thus, codes 
having to do with the reason for using the library come up when they are telling whether the 
results were helpful and even when they are discussing other ways in which they might get 
the needed information. 

Consequently, we made some major changes during the development of the coding 
scheme, so that a different scheme was used to code the responses to the question "What 
would you have done" after the first set of data had been analyzed. As described in Chapter 
4, we believe that it will be most useful to reorganize the extended set of codes (which are 
detailed in Appendix B) in terms of a conceptually structured taxonomy. At the present time, 
the mapping between codes and taxonomy is multivalued. That is, for each level in the 
taxonomy there are several corresponding codes, and several of the codes appear in 
conjunction with more than one of the items in the taxonomy. 

While we could, in principle, move directly to the new taxonomy by simply assigning 
a large number of codes as appropriate, we believe that the most productive path towards 
future research will be to completely recode these open-ended questions in terms of the new 
taxonomy, and to develop concise coding rules which support that. However, we see this as 
scholarly work falling outside the scope of the present project, and which will be undertaken 
at Rutgers as time permits. 

6.1.2 Calibrating individual coding labels 

We have developed a mathematical formalism which will make it possible to assess 
inter-judge consistency at the level of the individual label assigned. We believe that this will 
improve our ability to refine the taxonomy, by identifying those labels which prove difficult 
to assign consistently. 

6.1.3 Gathering better scale information 

As mentioned earlier, we have found that the lack of a more general context makes it 
difficult to differentiate the impact of several services with an instrument of this type. We 
speculate here on techniques that might be used to enhance the differentiating ability. 

One technique would be an extension of the critical incident method, in which 
respondents are asked to recall another recent use of the library and compare it with the 
present use of the library. This is, in a sense, an extension of conjoint analysis to the entirety 
of the library use event. It might prove possible to develop comparative measures of the 
impact of specific services by studying the frequency with which one service was judged to 
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have a greater impact than another. However, at the present this technique is speculative and 
might yield disappointing results. 

The second, rather more promising direction for extension is to consider the 
acquisition aspect of our conceptual triad. With regard to this, it should prove possible to 
frame a question such as "How long has it been between the moment when you first realized 
that you needed particular information for your project and the moment when you were able 
to get it from the library?". If this proves to be determinable by interview, it will help to 
distinguish various services, particularly those which are accessible online. However, like our 
earlier suggestion, this is still speculative and remains to be demonstrated in extended studies. 

6.2 Suggestions for application 

Suggestions for applications of this taxonomy to specific library services, or for 
experimentation, are embedded within the text, and will not be repeated. Section 4.3 suggests 
general approaches and methodologies from other fields that are applicable for applications in 
assessment of values of library services. Section 4.4 provides an outline of specific 
methodologies to be followed in development of similar taxonomies, together with 
suggestions for applications and generalizability of those methodologies. Section 4.5 presents 
a number of attributes of the taxonomy that should be a major consideration in any 
application. Section 4.7 presents elements and problems encountered in encoding, and these 
should enter in development of instructions in any application. Section 4.8 describes methods 
and results used in testing of the taxonomy - these can be used in any applications where our 
taxonomy is adjusted or a new one developed. Finally, Appendix B provides instruction used 
in this project for encoding using version 1 of the taxonomy. 

A given library can take the taxonomy developed here and the approaches used to 
undertake a study of its own. In such a case a generic six-stage process is recommended. 

1. Setting of objectives: Deciding what service(s) to study, what is the goal of the 
study, i.e. what should the study encompass, and why the study is carried out, 
i.e. what should the results be used for. Among others, results could be used 
for changes in services, marketing, justification, institution of additional 
services, design decisions, testing of services etc. 

2. Specifying methodologies: Develop specific methods to be used in the study. 
Includes: making adjustments, if any, to the taxonomy to fit specific situation 
and objectives; specifying specific procedures for collection and analysis of 
data; developing and testing of instalments; writing of manuals and 
instructions to carry out the procedures by different people; selecting samples, 
times, and places; and determining other application matters specific to the 
library and objectives. 

3. Allocating resources: Providing necessary financial, technical and human 
resources. Includes: establishing a budget; selecting professional and support 
personnel; training of personnel, particularly in relation to interviewing and 
encoding; establishing technical support, e.g. needed hardware and software; 
establishing a time, task and people schedule, and if necessary critical paths; 
and determining other needed resources specific to the library and objectives. 
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4. 



Data collection: Carrying out data collection procedures, according to 
established methodology. Testing them beforehand. 



5. Data analysis: Providing for a variety of analyses to fit the objectives. Data 
may lend itself to further analyses not envisioned at the outset. Both 
quantitative and qualitative analyses are applicable. 

6. Deriving conclusions: Interpret the meaning of data; connecting conclusions to 
objectives; extending interpretations and conclusions to pragmatic suggestions 
and recommendations; writing a report; presentations to management and 
various audiences. 

The detailed history of this project, Part 2 of this report, and the more concise Manual 
for replicating the study, Part 3 of this report will be helpful in following this general outline. 
In addition, the specific questions and coding schemes given in Appendices A-C will serve as 
a guide. However, we must warn potential adopters of these instruments that the process of 
developing the codes is time-consuming. 

A study of value of library services is a complex undertaking, and requires allocation 
of significant resources. It requires considerable thought and a team effort. Most of all, it 
requires a commitment to cooperate by both, management and staff. Such a study may be 
perceived by some of the staff or management as a personal threat. It is not, and it should not 
be. The study design must include highly visible safeguards to ensure that it is not used to 
"assess individuals". The study has to be carefully and repeatedly explained to all concerned. 
At most institutions, informed consent must be obtained from users who are interviewed. 

Libraries today face a number of problems and challenges: reduction or stagnation in 
financial resources; increase in competition from other information services, including local 
computer departments and local, national and international networks; strategic positioning vis- 
vis such services; application of new technologies and incorporation of electronic resources; 
provision of remote services "without walls"; challenges in responses to R&D in digital 
libraries and information infrastructure; and political struggles with internal services 
competing for limited resources. 

In the final analysis, people use libraries because they find value in them. In the past 
such value was considered as self-evident and widely understood and accepted. But changing 
times and new challenges require that libraries go beyond that in documenting their value to 
users. Study designs such as this one will help libraries to document the value that people 
find in libraries and express it in specific data that can be used to manage effectively in the 
face of these challenges. 



6.3 Conclusions 

We may sum up the results of our study by saying that it is definitely possible to 
obtain and classify user responses. In addition, it proves exceedingly difficult to untangle the 
three conceptual aspects: acquisition, cognition, and application in brief interview studies of 
library users. Correspondingly, it has proved difficult to obtain scale-based assessments 
which relate the present instance of library use to the general use of other library services, 
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■ and to the goals and value of the project which causes that library use. 

However, the present study has shown that it is definitely possible to develop a 
uniform instrument suitable for use at a variety of institutions to study a variety of services. 
We have also shown that it is possible to develop reasonably accurate unit cost estimates for 
diverse functions, in a way that does not require the library to develop a complete program 
budget. 

Finally, we have demonstrated how the techniques of Data Envelopment Analysis can, 
in principle, be applied to the study of the relation between library functional costs and the 
impact of the library services or functions. 



APLAB-B%D\CLR\FINALREV\FINALBIG.51 



67 

Copyright (c) 1995 



March 21, 1995 P. 66 



7 ACKNOWLEDGEMENTS 



This project would not have been possible without the cooperation of several 
colleagues in the project. Joann D'Esposito-Wachtmann served as Project Manager, 
supervising as many as 18 simultaneous interview studies in four geographic locations. In 
addition, she supervised the contracting of transcription and data entry services, and played a 
large role in the definition and testing of the successive instruments. She authored the second 
and third parts of this report, whose contents are indexed at the end of this document. 

Essential roles were also played by the students in the PhD program at SCILS: 
Michael Wilk, who developed the Empirical Taxonomy, Dong Li, who was responsible for 
moving data into the SPSS program, developing the data dictionary, and performing many 
preliminary analyses, and Allan Zhao, who developed many auxiliary programs to support the 
analysis of transcribed texts. 

Of course this project would not have been possible without the generous cooperation 
of the five libraries which were the subject of the study. Our thanks to Elaine Sloan and 
Carol Mandel at Columbia; Frank Polach, Marianne Gaunt, Margie Epp^ and Emily Fabiano 
at Rutgers; Carlton Rochell, Nancy Kranich, Lucinda Co vert- Vail, Nancy Friedland and 
Marlayna Gates at NYU; Donald Koepp, Nancy Klath, Mary Chaikin, Patty Gaspari-Bridges, 
David Goodman, and Janice Powell at Princeton; Joanne Harrar and Danuta Nitecki at 
Maryland, and Prof. Eileen Abels at the University of Maryland School of Library and 
Information Science. 

Financial support was provided by the Council on Library Resources through a grant 
to the Alexandria Project Laboratory at the School of Communication, Information and 
Library Studies at Rutgers, the State University of New Jersey. Additional support was 
provided by Dean Richard Budd of SCILS. Mr. Li and Mr. Wilk were supported as Graduate 
Research Assistants during their work on this study. 



6S 



o 

ERIC 



APLAB-B%D\CLR\FINAL.REV\FINALBI' 



Copyright (c) 1995 



March 21, 1995 P. 67 



8 REFERENCES AND LITERATURE 



We include here the references cited in the report, and a comprehensive bibliography 
of work on related issues of library measurement and evaluation. 

Bailey, Charles W.Jr. "Intelligent Multimedia Computer Systems: Emerging Information 
Resources in the Network Environment." Library Hi-Tech . 8:1 (1990): 29-41 

Bailey, K.D. (1994). Typologies and taxonomies. An introduction to classification techniques. 
Thousand Oaks, CA: Sage Publications. 

Baker, S.L., & Lancaster, F.W. (1991). The measurement and evaluation of library services. 
2nd Edition. Arlington, VA: Information Resources Press. 

Ahn.T., Charnes.A., and Cooper,W.W., "Using Data Envelopment Analysis to Measure the 
Efficiency of Not-For-Profit Organizations: A Critical Evaluation-Comment," 
Managerial and Decision Economics, Vol. 9, No. 3, (1988), pp. 251-253. 

Banker.R.D., Charnes.A., and Cooper.W.W., "Some Models for Estimating Technical and 
Scale Inefficiencies in Data Envelopment Analysis," Management Science, Vol. 30, 
No. 9, (1984), pp. 1078-1092. (The "BCC" Model) 

Bawden, D. (1991). User-oriented evaluation of information systems and services. Brookfield, 
VT: Gower Publishing Co. 

Belkin, N.J., Shan-Ju, C, Downs, T., Saracevic, T., and Shuynan, Z. "Taking Account of User 
Tasks, Goals and Behavior for the Design of Online Public Access Catalogs," 
Proceedings of the American Society for Information Science . 27:69-79. 

Buckland, Michael K. "Concepts of Library Goodness," Canadian Library Journal 39(2), 

1982. Reprinted in: Roberts, Stephen A. Costing and the Economics of Library and 
Information Services . London: ASLIB (The Association for Information Management; 
Reader Series Vol. 5), 1983, pp. 258-272. 

Carr, D. "The Agent and the Learner: Interactions in Assisted Adult Library Learning." Public 
Library Quarterly . 2(2):3-19, 1980. 

Carroll, Bonnie C: King, Donald W. "Value of Information," Drexel Library Quarterly 21:39- 
60, 1985. 

Charnes.A., Cooper.W.W., and Rhodcs,E., "Measuring the Efficiency of Decision Making 

Units, "European Journal of Operational Research, Vol. 2, No. 6, (1978). pp. 429-444. 
(The "CCR" Model) 

Chen, Ching-Chih. "As Wc Think: Thriving in the HyperWeb Environment." 
Microcomputers for Information Management . 6:2(Jun 1989): 77-97. 

Chinburapa V, Larson, LN. Predicting Prescribing Intention and Assessing Drug Attribute 
Importance Using Conjoint Analysis. Journal of Pharm Marketing and Management 



9 

:RIC 



APLAB-B%D\CI.R\FINAL.REV\FINALBIG.S1 Copyright (c) 1995 



March 21, 1995 P. 69 



63 



v3(2)p3- 18(1 988). • 

Corbin, J. and Strauss, S. "Grounded Theory Research: Procedures, Canons, and Evaluative 
Criteria" Qualitative Sociology , 13(1):3-21. 

Cummings, Martin M. The Economics of Research Libraries . Washington, D.C: Council on 
Library Resources, 1 986. 

Dervin, Brenda; Nilan, Michael. Information Needs and Uses. ARIST v21 p3-33 (1986) 

Feeney, M. & Grieves, M. (1994). The Value and Impact of Information . London: Bowker 
Saur. 

Flanagan, J.C. "The Critical Incident Technique." Psychological Bulletin , 51(4):327-58, 1954. 

Geiger, CJ; Wyse, BW; Parent, CRM; Hansen, RG. Nutrition Labels in Bar Graph format 

deemed most useful for consumer purchase decisions using adaptive conjoint analysis. 
J of the American Dietetic Association. v91(7)p800-807(1991). 

Getz, Malcolm. Public Libraries: an Economic View . Baltimore, Md: Johns Hopkins 
University Press, 1980. 

G-een, Paul E. "On the Design of Choice Experiments Involving Multifactor Alternatives". J 
of Consumer Research vlp61-68(1974). 

Green, PE, Srinivasan, V. Conjoint Analysis in Consumer Research: Issues and Outlook. J of 
Consumer Research v5pl03-123(Sept 1978). 

Green, PE and Wind, Y. "New Way to Measure Consumers Judgments" Harvard Business 
Review, v53pl07-l 17(1975). 

Green, PE, Srinivasan, V. Conjoint Analysis in Marketing: New Developments with 

Implications for Research and Practice. Journal of Marketing pp3- 19 (October 1990). 

Griffiths, J-M; King, DW. The Evaluation of Information Centers and Services , (to be 
published. The authors have kindly provided Chapter 1 1 to this project team.) 

Halperin, Michael. Determining User Preferences for Information Services. Drexel Library 
Quarterly. vl7 n2 p88-98 (Spring 1981). 

Halperin, Michael; Strazdon, Maureen. Measuring Students Preferences for Reference 
Services: A Conjoint Analysis. Library Quarterly v50 n2 p208-24. (Apr 1980). 

Hayes, Robert M. "The Management of Library Resources: the Balance Between Capital and 
Staff in Providing Services," Library Research 1:119-142, 1979. 

Hcwgill, .ICR. "Management Accounting and Library Activities," Aslib Proceedings 29(9), 
1977. R( printed in: Roberts, Stephen A. Costing and the Economics of Library and 
I nformation Services . London: ASLIB (The Association for Information Management; 



APLAB-B%D\CLR\FINAL.RF.V\FINALBIG.51 Copyright (c) 1995 



March 21. 1995 P. 70 



Reader Series Vol. 5), 1983, pp. 87-92. 



Hoenack, Stephen A.; Collins, Eileen L., editors. The Economics of American Universities; 
Management, Operations, and Fiscal Environment . Albany, N.Y.: State University of 
New York Press, 1990. 

Kantor, Paul B. BK1986a Costs of Preservation Microfilming at Research Libraries: A Study 
of Four Institutions . Council on Library Resources, 1986. 32pp. 

Kantor, Paul B. Rl985f "Implications of Formula Budgeting for the Growth of Library 

Services." Tantalus Technical Report TANTALUS/CT-85/3, August 30, 1985. 23pp. + 
Tables. 

Kantor, Paul B. J198 lh "Quantitative Evaluation of the Reference Process," RQ 
V21(l)p43-53, Fall 1981. 

Kantor, Paul B. Jl986c "Three Studies of Cost and Services at Academic Libraries," 
Advances in Library Administration and Management , V5p221-285, 1986. 

Kantor, Paul B. Rl989c "A Survey of Copy Cataloging Practices at ARL Libraries." Tantalus 
Technical Report TANTALUS/CT-89/1, 1989. (P. Kantor, M. Cherikh and S. Rich). 
llp.+ Appendices and Tables. 

Kantor, Paul B. Jl989a "Library Cost Analysis." In Problem Solving in Libraries, Library 
Trends V38(2)pl71-188, Fall 1989. 

Kantor, Paul B. J 1984b "Cost and Usage of Health Sciences Libraries: Economic Aspects," 
Bulletin of the Medical Library Association V72(3)p274-286, July 1984. 

Kantor, Paul B. R 1990a "Economic Aspects of the NCCP Pilot Project." Tantalus Technical 
Report TANTALUS/CT-90/1, 1990. 20pp. 

Kantor, Paul B. BKl986b Functional Cost Analysis for Libraries , Tantalus Inc, 1V86. 80pp. 

Kantor, Paul B. BK 1984a Objective Performance Measures for Academic and Research 
Libraries , The Association of Research Libraries, 1984. 76pp. + Exhibits. 

Kantor, Paul B. CH 1989b "Assessing the Costs During Budget Planning for Medline on CD- 
ROM." In Medline on CD-ROM , Learned Information Inc, Medford NJ, 1989, p35-43. 



Kantor, Paul B. "Levels of Output Related to Cost of Operation of Scientific and Technical 
Libraries," Library Research 3:1-28, 1981. 

Kantor, Paul B. R1985e "Output Measures and Public Library Standards: A Multidimensional 
Problem." Tantalus Technical Report TANTALUS/PR-85/3, July 1, 1985. 14pp. 

Kantor, Paul B. J 198 lc "Levels of Output Related to Cost of Operation of Scientific and 

Technical Libraries. Part I: Techniques and Cumulative Statistics," Library Research , 



APLAB-B%D\CLR\FINAL.REV\FINALBIG.S1 Copyright (c) 1 995 March 21, 1995 P. 71 



V3(l)pl-28, Spring 1981. 



Kantor, Paul B. B 1986a FUNCOST . A spreadsheet package for the functional analysis of 
library costs with manual of operations (80pp.). Available in three versions, for 
special, academic and public libraries. Tantalus Inc, 1986. 

Kantor, Paul B. J1978a "QUBMIS: A Quantitatively Based Management Information 

System." In Proceedings of the American Society for Information Science, pl74-176, 
1978. 

Kantor, Paul B. J1981f "Levels of Output Related to Cost of Operation of Scientific and 

Technical Libraries, Part II: A Capacity Model of the Average Cost Formula," Library 
Research V3(2)p 141 -154, Summer 1981. 

Kantor, Paul B. "Levels of Output Related to Cost of Operation of Scientific and Technical 

Libraries: Part II, A Capacity Model of the Average Cost Formula," Library Research 
3:141-154, 1981. 

Kantor, Paul B. "A Review of Library Operations Research," Library Research 1:295-345, 
1979. 

Kent, Allen; Cohen, Jacob; Montgomery, K. Leon, issue editors. The Economics of 

Academic Libraries . Urbana, 111: University of Illinois; entire issue of: Library Trends , 
Vol. 28, No. 1, Summer 1979. 

King, DW; Griffiths, J-M. "Evaluating the Effectiveness of Information Use". In Evaluating 

the Effectiveness of Information Centers and Services : AGARD Lecture Series No. 60. 
NATO AGARD-LS-160. 

Koenig, Michael E. D. Budgeting Techniques for Libraries and Information Centers . New 
York: Special Libraries Association (Professional Development Series, Vol. One), 
1980. 

Lederman, L.C. "Assessing Educational Effectiveness: The Focus Group Interview as a 
Technique for Data Collection.' 1 Communication Education . 117-127, 1990. 

Li, Tze-chung. "Information Retrieval in Multimedia Sources in an Electronic Age." ERIC 
Clearinghouse on Information Resources , (1988), 19p. 

McGrath, William E. "A Pragmatic Book Allocation Formula for Academic and Public 

Libraries with a Test for its Effectiveness," Library Resources & Technical Services 
19(4):356-369. 

Mick, C. K. "Cost Analysis of Information Systems and Services," Annual Review of 
Information Science and Technology 14:37-64, 1979. 

Mitchell, Betty Jo; Tunis, Norman E.; Jaffe, Jack. Cost Analysis of Library Functions . 
Greenwich, Conn.: JAI Press, 1978. 



7 9 

APLAB-B%D\CLR\FINAL.REV\FINALBIG.51 Copyright (c) 1995 ' C March 21, 1995 P. 72 



Nickerson CAE, McClelland, GH, Petersen, DM. Solutions to some problems in the 

implementation of Conjoint Analysis. Behaviors Research Methods, Instruments and 
computers v22(4)p36()-374(1990). 



Orr, R.H. "Measuring the Goodness of Lib'ary Services: a General Framework for 

Considering Quantitative Measures." Journal of Documentation 29(3), 1973. Reprinted 
in: Roberts, Stephen A. Costing and the Economics of Library and Information 
Services . London: ASLIB (The Association for Information Management; Reader 
Series Vol. 5), 1983, pp. 240-257. 

Raitt, David. "The Potential of New Technology in Information Environments." 
Microcomputers for Information Management . 6:2(Jun 1989): 99-112. 

Ramsing, Kenneth D.; Wish, JR. What Do Library Users Want? A Conjoint Measurement 

Technique May Yield the Answer. Inf. Proc. and Management vl8 n5 p237-42 (1982). 

Roberts, Stephen A. Cost Management for Libraries and Information Services . London, 
Butterworths, 1984. 

Repo, A.J. (1989). The value of information: Approaches in economics, accounting, and 
management science. Journal of the American Society for Information Science, 40, 
(2),68-85. 

Roberts, Stephen A. Costing and the Economics of Library and Information Services. 

London: ASLIB (The Association for Information Management; Reader Series Vol. 5), 
1983. 

Rosenberg, P. Cost Finding for Public Libraries: A Manager's Handbook. Chicago. American 
Library Association. (1985) 

Ruben, B.D., "The Health Caregiver-Patient Relationship: Pathology, Etiology, Treatment." In 
Communication and Health: Systems and Applications (E.B. Ray and L. Donohew, 
eds.), Hillsdale, NJ: Lawrence Erlbaum, 1989, pp. 51-68. 

Ruben, B.D., Communicating with Patients . Dubuque, IA: Kendall-Hunt, 1992, 116pp. 

Schamber, Linda. "Technology Options for Libraries." ERIC Clearinghouse on Information 
Resources . (May 1988), 3p. 

Seiford.L.M., "A Bibliography of Data Envelopment Analysis", Technical Report, Department 
of Industrial Engineering and Operations Research, University of Massachusetts at 
Amherst, (1978-1990). 

Taylor, R.S. (1986). Value-added processes in information systems. Norwood, NJ: Ablex 
Publishing Co. 

Weber, R.P. (1990). Basic content analysis. 2nd cd. Newbury Park, CA: Sage Publications. 
Shera, J. H. T he Foundation of Education for Librarianship . Wiley, New York, 1972. 



APLAB-B%D\CLR\FINAL.REV\FINALBIG.51 Copyright (c) 1995 



73 



March 21, 1995 P. 73 



Smith, G. Stevenson. "Managerial Accounting and Changing Models of Administrative 

Behavior: New Methods for New Models," Library Trends 38(2): 189-203, Fall 1989. 



Taylor, Robert S. Value-added processes in information systems . Norwood, N.J. Ablex Pub. 
Corp., cl986. 

Turock, Betty J., Editor, The Bottom Line: A Financial Magazine for Librarians . New York: 
Neal-Schuman, quarterly. 

Turock, Betty J., Creating A Financial Plan New York: Neal Schuman, in press. 

Turock, Betty J., Improving Library Program Evaluation . Washington, D.C.: U.S. 

Department of Education, Office of Library Research and Improvement, Library 
Programs, 1992. 

Turock, Betty J., Editor, Evaluating Federally Funded Library Programs . Washington, D.C.: 
U.S. Department of Education, Office of Library Research and Improvement, Library 
Programs, 1991. 

Umesh, UN, Mishra, S. "A Monte-Carlo Investigation of Conjoint Analysis Index of Fit: 
Goodness of Fit, Significance and Power", Psychometrika v55(l)p33-44(1990). 

Vasi, John. Budget Allocation Systems for Research Libraries . Washington, D.C.: 

Association of Research Libraries (Office of Management Studies; Occasional Paper 
No. 7), 1983. 

Virgo, Julie A.C. "Costing and Pricing Information Services," Drexel Library Quarterly 23:75- 
98, 1987. 

Waterhouse, Lloyd G. "A Vision of Technology for Higher Education." Library 
Administration & Management . 5:2(Spr 1991):89-92. 

Whitehall, T. "User Valuations and Resource Management for Information Services," Aslib 
Proceedings 32(2), 1980. Reprinted in: Roberts, Stephen A. Costing and the 
Economics of Library and Information Services . London: ASLIB (The Association for 
Information Management; Reader Series Vol. 5), 1983, pp. 273-291. 

Wilson, J. H. "Costs, Budgeting, and Economics of Information Processing,'" Annual Review 
of Information Science and Technology 7:39-67, 1972. 

Wilson, Sandra R., Starr-Schneidkraut, Norma and Cooper, Michael D. Use of the Critical 
Incident Technique to Evaluate the Impact of MEDLINE American Institute for 
Research NLM contract no. Nol- LM-8-3529, 1989. 



74 



APLAB-B%D\CLR\FINALREV\FINALBIG.51 Copyright (c) 1995 



March 21, 1995 P, 74 



9 Contents: Volume 2: Process 



1 Table of Contents 2 

2 INTRODUCTION 7 

3 PRELIMINARY STUDIES AND WAVES 1.1 and 1.2 8 

3.1 Schedule of Wave 1 8 

3.2 The Libraries and the Services Studied 8 

3.2.1 Service Selection 8 

3.2.2 LIBRARY 1 8 

3.2.2.1 Service 1 8 

3.2.2.2 Service 2 9 

3.2.2.3 Service 3 9 

3.2.3 LIBRARY 2 9 

3.2.3.1 Service 1 9 

3.2.3.2 Service 2 10 

3.2.3.3 Service 3 , 10 

3.2.4 LIBRARY 3 10 

3.2.4.1 Service 1 10 

3.2.4.2 Service 2 10 

3.2.4.3 Service 3 10 

3.2.5 LIBRARY 4 10 

3.2.5.1 Service 1 11 

3.2.5.2 Service 2 11 

3.2.5.3 Service 3 11 

3.2.6 Library 5 11 

3.2.6.1 Service 1 11 

3.2.6.2 Service 2 11 

3.2.6.3 Service 3 11 

3.3 Quotas and Response Rates: Wave I 12 

3.3.1 Site Visits 12 

3.3.2 Establishing Quotas 12 

3.3.3 Study Outcome: Response Rates 12 

3.4 Details of Survey Administration By Service 21 

3.4.1 LI REFERENCE SERVICE 21 

3.4.1.1 Information Desk 21 

3.4.1.2 Reference Desk 21 

3.4.1.2.1 Branch A (21); 3.4.1.2.2 Branch B (21) 

3.4.1.3 Reference Consultation 21 

3.4.2 LI: MATERIALS DELIVERY SERVICE (MDS) 21 

3.4.3 LI: BIBLIOGRAPHIC INSTRUCTION 22 

3.4.4 L2 : ART AND ARCHAEOLOGY - THE COLLECTION 22 

3.4.5 L2: BIOLOGY - ELECTRONIC REFERENCE 22 

3.4.6 L2: PSYCHOLOGY - REFERENCE 22 

3.4.7 L3: ENHANCED ONLINE CATALOG 23 

3.4.7.1 On-site 23 

3.4.7.2 Online 23 

3.4.8 L3: UNDERGRADUATE RESERVE 23 



APLAB-B%D\CLR\FINAL.REV\FINALBIG.51 Copyright (c) 1995 



'5 



March 21, 1995 P. 75 



Rir 



3.4.8.1 Faculty 23 

3.4.8.2 Students 24 

3.4.9 L3: SCIENCE LIBRARIES 24 

3.4.9.1 Document Delivery 24 

3.4.10 L3: Carl UNCOVER 24 

3.4.11 L4: ELECTRONIC REFERENCE 24 

3.4.12 L3: MUSIC AND MEDIA CENTER 25 

3.4.12.1 On-site Users 25 

3.4.12.2 Faculty 25 

3.4.13 L3: INTERLIBRARY LOAN ... 25 

3.4.14 L5: AUTOMATED REFERENCE 25 

3.4.15 L5: REFERENCE SERVICE 25 

3.4.16 L5: PATENT SERVICE 26 

3.5 THE INTERVIEWERS 26 

3.5.1 Recruiting Interviewers 26 

3.5.2 Interviewing the Applicants 26 

3.5.3 Training 27 

3.5.3.1 The agenda for the training session was as follows: 27 

3.5.3.1.1 Welcome: 5 minutes (27); 3.5.3.1.2 



Questionnaire Familiarization - 10 minutes (28); 3.5.3.1.3 
Interviewing- 45 minutes (28); 3.5.3.1.4 Break - 5 
minutes (28); 3.5.3.1.5 Questionnaire/Instructions/Tally 
Sheet - 30 minutes (28); 3.5.3.1.6 General 
Rules/Information - 20 minutes (29); 3.5.3.1.7 Practice 
Interviewing - 10 minutes (29); 3.5.3.1.8 Closing (29) 



3.5.3.2 Training at Library L5 30 

3.5.4 Reviewing the Work of Interviewers 30 

3.6 THE RESEARCH INSTRUMENTS 31 

3.6.1 Focus Groups 31 

3.6.2 The Questionnaire 31 

3.6.2.1 LI, L2, L3, L4 (Version 1.1) 31 

3.6.2.2 Instrument at Library L5 (Version 1.2) 32 

3.7 PROJECT MANAGEMENT 33 

3.7.1 Human Subjects Protection 33 

3.7.2 Scheduling Interviewers 33 

3.7.2.1 In-Person Interviews 33 

3.7.2.2 Telephone Interviewers 33 

3.7.2.3 Interviewer Numbers 34 

3.7.2.4 Equipment/Supplies 34 

3.7.2.5 Numbering the Questionnaires 34 

3.7.2.6 Tally Sheets 34 

3.7.2.7 Call Disposition Sheets 35 

3.7.2.8 Editing 35 

3.7.2.9 Keypunching/Transcribing 35 

3.7.2.9.1 Keypunching (35); 3.7.2.9.2 Transcribing (36) 

3.7.2.10 Labor Costs 36 

4 DELAYED IMPACT ASSESSMENT: WAVE 2 37 

4.1 Schedule of Wave 2 37 

APLAB-B%D\CLR\FINAL.REV\FINALBIG.51 Copyright (c) 1995 ^ March 21, 1995 P. 76 



4.2 THE SERVICES 37 

4.2.1 Library LI 37 

4.2.2 Library L2 37 

4.2.3 Library L3 37 

4.2.4 Library L4 38 

4.2.5 Library L5 38 

4.3 Recontacting Librarians 38 

4.4 Establishing Quotas 38 

4.5 Study Outcome: Response Rates 38 

4.5.1 LI: Reference Service 46 

4.5.1.1 Information Desk 46 

4.5.2 LI: Reference Desk 46 

4.5.2.1 Branch A 46 

4.5.2.2 Branch B 46 

4.5.3 LI: Reference Consultation 46 

4.5.4 LI: MATERIALS DELIVERY SERVICE 46 

4.5.5 L2: ART AND ARCHEOLOGY - THE COLLECTION 46 

4.5.6 L2: BIOLOGY - ELECTRONIC REFERENCE 46 

4.5.7 L2: PSYCHOLOGY - REFERENCE 46 

4.5.8 L3: ENHANCED ONLINE CATALOG 46 

4.5.9 L3: SCIENCE LIBRARIES 47 

4.5.10 L4: ELECTRONIC REFERENCE 47 

4.5.11 L4: INTERLIBRARY LOAN 47 

4.5.12 L5: AUTOMATED REFERENCE 47 

4.5.13 L5: REFERENCE SERVICE 47 

4.5.14 L5: PATENT SERVICE 47 

4.6 THE INTERVIEWERS 48 

4.6.1 Recruiting Interviewers 48 

4.6.2 Training 48 

4.6.3 Reviewing 48 

4.7 THE RESEARCH INSTRUMENT 49 

4.7.1 The Questionnaires 49 

4.8 PROJECT MANAGEMENT 50 

4.8.1 Scheduling Interviewers 50 

4.8.2 Numbering the Questionnaires 50 

4.8.3 Color Coding 50 

4.8.4 Tally Sheets 50 

4.8.5 Call Disposition Forms 50 

4.8.6 Tracking Callback and Phone Interviews 51 

4.8.7 Keypunching 51 

4.8.8 Editing 51 

4.8.9 Transcribing 52 

5 ACKNOWLEDGEMENTS 53 



77 



APLAB-B%D\CLR\FINAL.REV\FINALBIG.51 Copyright (c) 1995 March 21, 1995 P. 77 

er|c 



Index of Tables 



Table 1 Wave I Projected Quotas 14,15 

Table 2 Wave I Completed Interviews 16,17 

Table 3 Wave I Completes Per Hour 18 

Table 4 Wave I In-person Interviews-Contact Dispositions 19,20 

Table 5 Wave I Contact Dispositions-Telephone Interview 21 

Table 6 Wave II Projected Quotas 41 

Table 7 Wave II Completed Interviews 42 

Table 8 Wave II Completes Per Hour 43 

Table 9 Wave II Contact Dispositions-Audition 44 

Table 10 Wave II Contact Dispositions-Callbacks 45 

Table 1 1 Wave II Contact Dispositions-Phone Interview 46 

Table 12 Wave II Percentage of Auditions Resulting in Callbacks 47 



7S 



APLAB-B%D\CLR\FINAL.REV\FINALBIG.51 Copyright (c) 1995 



March 21, 1995 P. 78 



10 CONTENTS: Volume 3: MANUAL 



TABLE OF CONTENTS: THIS MANUAL 2 

PREPARATION FOR A STUDY 4 

2.1 Recruitment 4 

2.2 Master schedule 5 

2.3 Hiring interviewers 6 

THE METHODOLOGY/THE QUESTIONNAIRE 8 

3.1 Selection of Subjects. Customization of Instruments 8 

3.2 The Audition Interview 8 

3.3 The Callback Interview 9 

3.4 The Phone Interview 9 

3.5 A Note about the Questionnaire 9 

QUESTIONNAIRE INSTRUCTIONS 11 

PROJECT MANAGEMENT 12 

5.1 Human Subjects Protection 12 

5.2 Scheduling Interviewers 12 

5.2.1 Equipment; Supplies 12 

5.2.2 Tally She-.ts 13 

5.3 Reviewing 13 

5.3.1 Editing 14 

5.3.2 Telephone Interviewing 14 

5.3.3 Tracking, Callback and Phone Interviews 14 

5.3.4 Keypunching 14 

5.3.5 Transcribing 15 

TRAINING 16 

6.1 16 

6.2 Questionnaire Familiarization - 10 minutes 16 

6.3 Interviewing - 45 minutes 16 

6.4 Break - 5 minutes 17 

6.5 Questionnaire Instructions/Tally Sheet - 30 minutes 17 

6.5.1 General Rules/Time Sheet - 20 minutes 17 

6.5.2 Practice Interviewing - 10 minutes 18 

6.5.3 18 

CODING RULES FOR DEPARTMENT/AREA STUDIES 19 

ACKNOWLEDGEMENTS 21 



79 

APLAB-B%D\CLR\FINAL.REV\FINALBIG.S1 Copyright (c) 1995 March 21, 1995 P. 79 

er|c 



1 1 Contents of Appendices 



APPENDIX A 



A.l. Wave 1 
A.2. Wave 1 
A.3. Wave 2 
A.4. Wave 2 
A.5. Wave 2 



First Version 

Revised Version (Maryland) 

Audition 

Callback 

Phone 



A.6. Mapping of SPSS variables to questions 



APPENDIX B 



Open Ended Coding 



Bl. Map of Codes to Questions 

B2. Map of Codes used to Proposed Codes 



Bl 
B8 



APPENDIX C 

Open Ended Coding Result Tables 

APPENDIX D 

Cost Data Collection Forms 

Dl. Cover memo 

D2. Extracts from Tantalus Inc FUNCOST Manual 
D3. White Forms: Personnel and Other Direct Costs 
D4. Yellow Form. Estimate of Service Rendered 
D5. Return Instructions. 

APPENDIX E 

Interviewer Training Manual 

APPENDIX F 

Miscellaneous Forms 

F. 1 . Posting to recruit interviewers 

F.2. Letter to invite applicant 

F.3. Application form Wave 1 

F.4. Application form Wave 2 

F.5. Letter of appointment 

F.6. Interviewing Schedule 

F.7. Interviewer Tally Sheet 

F.8. Telephone interview Tally Sheet 

F.9. Letter at end of project g q 

F.10. Telephone interview schedule 

APLAB-B%D\CLR\FINAL.REV\FINALBIG.S1 Copyright (c) 1995 March 21, 1995 P. 81 



> F.ll. Call disposition form - Wave 2 
F.12. Site Feasibility Report 
F.13. Interviewer time sheet 
F.14. Examples of contact memos or messages 

APPENDIX G 

Focus Group Moderator Guide 

APPENDIX H 
Questionnaire Instructions 

H.l. Wave 1 

H.2. Wave 2 Audition 

H.3. Wave 2 Callback 

H.4. Wave 2 Phone 

H.5. General Telephone instructions 

H.6. Site placement instructions 



81 



APLAB-B%D\CLR\FINAL.REV\FINALBIG.51 Copyright (c) 1995 



March 21, 1995 P. 82 



■ERIC 



Cost and Beneficial Impacts 
of Library Services 

Part 2: Detailed History of Project 
Management and Processes 

by 

Joann D'Esposito-Wachtmann 
Project Manager 

Paul B. Kantor 
Tefko Saracevic 
Principal Investigators 



Alexandria Project Laboratory. School of Communication Information and Library Studies, 
Rutgers, the State University of New Jersey. 

Technical Report APLAB/94-3/2. (Rev Mar 1995) 

Based on Research Supported in Part by the Council on Library Resources. 



APLAB-B%D\CLR\FINAL.REV\PROCESS.51 Copyright (c) 1995 March 21, 1995 P. 1 

O 

ERIC 

UMMMtatmm 



1 Table of Contents 



1 Table of Contents 2 

2 INTRODUCTION 7 

3 PRELIMINARY STUDIES AND WAVES 1.1 and 1.2 8 

3.1 Schedule of Wave 1 8 

3.2 The Libraries and the Services Studied 8 

3.2.1 Service Selection 8 

3.2.2 LIBRARY 1 8 

3.2.2.1 Service 1 8 

3.2.2.2 Service 2 9 

3.2.2.3 Service 3 9 

3.2.3 LIBRARY 2 ." ; 9 

3.2.3.1 Service 1 9 

3.2.3.2 Service 2 10 

3.2.3.3 Service 3 10 

3.2.4 LIBRARY 3 10 

3.2.4.1 Service 1 10 

3.2.4.2 Service 2 10 

3.2.4.3 Service 3 10 

3.2.5 LIBRARY 4 10 

3.2.5.1 Service 1 11 

3.2.5.2 Service 2 11 

3.2.5.3 Service 3 11 

3.2.6 Library 5 11 

3.2.6.1 Service 1 11 

3.2.6.2 Service 2 11 

3.2.6.3 Service 3 11 

3.3 Quotas and Response Rates: Wave I 12 

3.3.1 Site Visits 12 

3.3.2 Establishing Quotas 12 

3.3.3 Study Outcome: Response Rates 12 

3.4 Details of Survey Administration By Service 21 

3.4.1 LI REFERENCE SERVICE 21 

3.4.1.1 Information Desk 21 

3.4.1.2 Reference Desk 21 

3.4.1.2.1 Branch A (21); 3.4.1.2.2 Branch B (21) 

3.4.1.3 Reference Consultation 21 

3.4.2 LI: MATERIALS DELIVERY SERVICE (MDS) 21 

3.4.3 LI: BIBLIOGRAPHIC INSTRUCTION 22 

3.4.4 L2 : ART AND ARCHAEOLOGY - THE COLLECTION 22 

3.4.5 L2: BIOLOGY - ELECTRONIC REFERENCE 22 



APLAB-B%D\CLR\FINALREV\PROCESS.51 



Copyright (c) 1995 



March 21, 1995 P. 2 



3.4.6 L2: PSYCHOLOGY - REFERENCE 22 

3.4.7 L3: ENHANCED ONLINE CATALOG 23 

3.4.7.1 On-site 23 

3.4.7.2 Online 23 

3.4.8 L3: UNDERGRADUATE RESERVE 23 

3.4.8.1 Faculty 23 

3.4.8.2 Students 24 

3.4.9 L3: SCIENCE LIBRARIES 24 

3.4.9.1 Document Delivery 24 

3.4.10 L3: Carl UNCOVER 24 

3.4.11 L4: ELECTRONIC REFERENCE 24 

3.4.12 L3: MUSIC AND MEDIA CENTER 25 

3.4.12.1 On-site Users 25 

3.4.12.2 Faculty 25 

3.4.13 L3: INTERLIBRARY LOAN 25 

3.4.14 L5: AUTOMATED REFERENCE 25 

3.4.15 L5: REFERENCE SERVICE 25 

3.4.16 L5: PATENT SERVICE 26 

3.5 THE INTERVIEWERS 26 

3.5.1 Recruiting Interviewers 26 

3.5.2 Interviewing the Applicants 26 

3.5.3 Training 27 

3.5.3.1 The agenda for the training session was as follows: 27 



3.5.3.1.1 Welcome: 5 minutes (27); 3.5.3.1.2 
Questionnaire Familiarization - 10 minutes (28); 3.5.3.1.3 
Interviewing- 45 minutes (28); 3.5.3.1.4 Break - 5 minutes 
(28); 3.5.3.1.5 Questionnaire/Instructions/Tally Sheet - 30 
minutes (28); 3.5.3.1.6 General Rules/Information - 20 
minutes (29); 3.5.3.1.7 Practice Interviewing - 10 minutes 



(29); 3.5.3.1.8 Closing (29) 

3.5.3.2 Training at Library L5 30 

3.5.4 Reviewing the Work of Interviewers 30 

3.6 THE RESEARCH INSTRUMENTS 31 

3.6.1 Focus Groups 31 

3.6.2 The Questionnaire 31 

3.6.2.1 LI, L2, L3, L4 (Version 1.1) 31 

3.6.2.2 instrument at Library L5 (Version 1.2) 32 

3.7 PROJECT MANAGEMENT 33 

.. 7.1 Human Subjects Protection 33 

3.7.2 Scheduling Interviewers 33 

3.7.2.1 In-Person Interviews 33 

3.7.2.2 Telephone Interviewers 33 

3.7.2.3 Interviewer Numners 34 

3.7.2.4 Equipment/Supplies 34 

APLAB-B%D\CLR\FINAL.REV\PROCESS.51 Copyright (c) 1 995 March 21, 1995 P. 3 

ERIC 

ummmmmim 



3.7.2.5 Numbering the Questionnaires 34 

3.7.2.6 Tally Sheets 34 

3.7.2.7 Call Disposition Sheets 35 

3.7.2.8 Editing 35 

3.7.2.9 Keypunching/Transcribing 35 

3.7.2.9.1 Keypunching (35); 3.7.2.9.2 Transcribing (36) 

3.7.2.10 Labor Costs 36 

4 DELAYED IMPACT ASSESSMENT: WAVE 2 37 

4.1 Schedule of Wave 2 37 

4.2 THE SERVICES 37 

4.2.1 Library LI 37 

4.2.2 Library L2 37 

4.2.3 Library L3 37 

4.2.4 Library L4 38 

4.2.5 Library L5 38 

4.3 Recontacting Librarians 38 

4.4 Establishing Quotas 38 

4.5 Study Outcome: Response Rates 38 

4.5.1 LI: Reference Service 46 

4.5.1.1 Information Desk 46 

4.5.2 LI: Reference Desk 46 

4.5.2.1 Branch A 46 

4.5.2.2 Branch B 46 

4.5.3 LI: Reference Consultation 46 

4.5.4 LI- MATERIALS DELIVERY SERVICE 46 

4.5.5 L2: ART AND ARCHEOLOGY - THE COLLECTION 46 

4.5.6 L2: BIOLOGY - ELECTRONIC REFERENCE 46 

4.5.7 L2: PSYCHOLOGY - REFERENCE 46 

4.5.8 L3: ENHANCED ONLINE CATALOG 46 

4.5.9 L3: SCIENCE LIBRARIES 47 

4.5.10 L4: ELECTRONIC REFERENCE 47 

4.5.11 L4: INTERLIBRARY LOAN 47 

4.5.12 L5: AUTOMATED REFERENCE 47 

4.5.13 L5: REFERENCE SERVICE 47 

4.5.14 L5: PATENT SERVICE 47 

4.6 THE INTERVIEWERS 48 

4.6.1 Recruiting Interviewers 48 

4.6.2 Training 48 

4.6.3 Reviewing 48 

4.7 THE RESEARCH INSTRUMENT 49 

4.7.1 Tne Questionnaires 49 

4.8 PROJECT MANAGEMENT 50 

4.8.1 Scheduling Interviewers 50 



APLAB-B%D\CLR\FINALREV\PROCESS.51 Copyright (c) 1995 March 21, 1995 P. 4 



er|c " J 

UMMimmmM 



4.8.2 Numbering the Questionnaires 50 

4.8.3 Color Coding 50 

4.8.4 Tally Sheets 50 

4.8.5 Call Disposition Forms 50 

4.8.6 Tracking Callback and Phone Interviews 51 

4.8.7 Keypunching 51 

4.8.8 Editing 51 

4.8.9 Transcribing 52 



ACKNOWLEDGEMENTS 53 



APLAB-B%D\CLR\FINALREV\PROCESS.51 



er|c 



Copyright (c) 1995 



March 21, 1995 P. 5 



Index of Tables 

Table 1 Wave I Projected Quotas 14,15 

Table 2 Wave I Completed Interviews 16,17 

Table 3 Wave I Completes Per Hour 18 

Table 4 Wave I In-person Interviews-Contact Dispositions 19,20 

Table 5 Wave I Contact Dispositions-Telephone Interview 21 

Table 6 Wave II Projected Quotas : 41 

Table 7 Wave II Completed Interviews 42 

Table 8 Wave II Completes Per Hour 43 

Table 9 Wave II Contact Dispositions-Audition 44 

Table 10 Wave II Contact Dispositions-Callbacks 45 

Table 1 1 Wave II Contact Dispositions-Phone Interview 46 

Table 12 Wave II Percentage of Auditions Resulting in Callbacks 47 



APLAB-B%D\CLR\FINAL.REV\PROCESS.51 Copyright (c) 1995 March 21, 1995 P. 6 



ERIC 



2 INTRODUCTION 



To measure the benefits of library services, we conducted a three-phase interview- 
based study in five university research libraries. The sample focused on faculty members and 
graduate students in that order of priority but, numerically, graduate students were more 
highly represented than faculty. 

The preliminary qualitative phase explored, in a focus group setting, the vocabulary used 
to describe perceptions of library benefits. We assembled two focus groups: one consisting of 
faculty users of the libraries and one with graduate students. Results were incorporated into 
the later two quantitative phases. 

In the later two phases, called Wave I and Wave II, we sought a quantitative picture of 
the perceived measures of library benefits. Wave I concentrated upon those impacts which 
can be perceived and articulated by library users as soon as they have finished receiving the 
library service. Wave II explored the longer term impact of the information gained and its 
relation to the missions of the individual and of the institution as a whole. 

The interviews were conducted by telephone or in-person, as appropriate to the nature 
of the service. 
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3 PRELIMINARY STUDIES AND WAVES 1.1 and 1.2 

3.1 Schedule of Wave 1 



1 1/29-12/1 1/93 Interview-Weeks 2-3* 
*We skipped Thanksgiving week. 

Overall, wMwwere able to keep to the schedule with one exception. The fifth 
institution, l ibrary 5 (L5), was recruited later than Libraries 1-4 (LI - L4) and thereafter, 
interviews at that institution took place later than at the others (2/21/94-3/1 1/94). 

Data collection was managed directly from the APLab or with the participation of a library 
school faculty member who managed the study under our guidance. 

3.2 The Libraries and the Services Studied 

3.2.1 Service Selection 

We asked each library io select three specific services to study, using these suggested 
guidelines: 

A) A reference service in which the user is helped by a human member of the 
library staff in order to get the necessary information; 

B) An access service in which the user proceeds through online catalogs and 
other finding tools to locate the materials that are needed; 

C) A specialized subcollection; or 

D) Some other service that would be useful or helpful to the particular library. 
Their selections are presented below. 

3.2.2 LIBRARY 1 (LI) 
3.2.2.1 Service 1 

As its first selection, LI chose the thrcc-ticrcd Reference Service and requested that 
we study each of the tiers individually as follows: 
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We established an approximate schedule for Wave I as follows: 



9/7-30/93 
10/1-30/93 
11/1-13/93 
11/15-19/93 



Recruit Interviewers/Libraries Select Services 
Conduct Focus Groups/Develop Questionnaire 
Pretest Questionnaire/Train Interviewers 
Interview-Week 1 



ERIC 



o J 



The Information Desk 

This is the first "tier" of the Reference Service. At a desk situated near the 
library entrance, a library staff member addresses "simple" reference questions. 



The Reference Desk 

This is the second tier of the Reference Service; the user asks the reference 
librarian a question at the reference desk and the reference librarian responds to 
the query at that time. LI requested we study this service at two different 
library units. 

Reference Consultation Service 

This third tier supports users who want extensive reference assistance. The 
users complete request forms and arrange appointments with a reference 
librarian, who provides an in-depth consultation. 

3.2.2.2 Service 2 

As its second selection, LI chose a Materials Delivery Service (MDS) through which 
users can have items from one of the campus libraries delivered to whichever other library 
site is most convenient for them. Users request items either by filling out a request form at 
the circulation desk of the library to which they would like the items delivered or by 
submitting the request electronically through the campus-wide INFO Service. 
LI requested we study both types of users. This service was chosen, in part, in order 

to include university libraries on all of the three university campuses (which are located in 
different cities). 

3.2.2.3 Service 3 

For its third service, LI chose Bibliographic Instruction (BI) in which university 
professors bring their classes to the library for orientation and training. LI wanted to 
evaluate the benefit of BI by comparing perceptions of those students who have participated 
in it with those who have not. 

3.2.3 LIBRARY 2 - (L2) 

L2 responded to our request for service selection by focusing upon three of it3 smaller 
libraries: Art and Archeology, Biology, and Psychology, and selecting a service provided by 
each one. 

3.2.3.1 Service 1 Art and Archeology Library 

This library contains a non-circulating collection and L2 chose to study use of the 
collection itself. 
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3.2.3.2 Service 2 Biology Library 



In this library, L2 chose to focus upon use of the electronic reference services 
including MEDLINE, Science Citation Inci^x, Current Contents, Carl UNCOVER, the online 
catalog, LEXIS/NEXIS, the Internet, Biological, Abstracts, etc. 

3.2.3.3 Service 3 Psychology Library 

In this library, L2 selected the traditional in-person Reference Service. 

3.2.4 LIBRARY 3 - (L3) 

3.2.4.1 Service 1 

For its first service, L3 selected the enhanced online catalog. In addition to general 
library information and hours, and an electronic encyclopedia, this also contains the law 
library catalog, periodical indexes, catalogs from other libraries in the surrounding area and 
national bibliographic databases. L3 requested we interview both on-site users and remote 
users of this service. 

3.2.4.2 Service 2 

For its second service, L3 selected the undergraduate Reserve Service. (Although our 
focus was on faculty and graduate students, we made an exception for this service on the 
grounds that it is a service to faculty, and to their educational mission). Professors place 
course materials on reserve and students may borrow them and use them within the library for 
up to two hours. After 3:00 p.m., the materials may be borrowed u Kir 10 a.m. the following 
day. L3 requested we study both the faculty who had placed items on reserve and the 
students who use the service. 

3.2.4.3 Service 3 

For its third selection, L3 chose two types of services provided by the science 
libraries. The first is the Document Delivery Service at the Biology Library. Users submit 
requests for document delivery at the circulation desk; the librarians fill the requests; and 
users either retrieve the documents personally or have them sent by mail. 

The second type of service is a pilot Carl UNCOVER project in which the science 
faculty members are given their own Carl UNCOVER account to order documents as they 
deemed appropriate, rather than having to go through the librarian. 



3.2.5 LIBRARY 4 - (L4) 
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3.2.5.1 Service 1 

For its first service, L4 chose the Electronic Reference Resources. L4's library is a 
multi-story building with reference areas on three floors. Each provides different electronic 
sources, depending upon the subject matter housed there. L4 requested that we study users 
on all three floors. 

3.2.5.2 Service 2 

L4's second service is a Music and Media Center which functions similarly to a 
Reserve Service. Instructors place material on reserve, users come to listen to, or watch, 
tapes. Video users are assigned a carrel and the film is controlled from behind the counter. 
Music cassettes are handed to the user and may be borrowed for a specified period of time. 
Users are not permitted to remove materials from the Center. An exception is made for 
instructors who may borrow an item for classroom use and then return it. L4 requested we 
study both faculty who had placed items on reserve and on-site users. 

3.2.5.3 Service 3 

For its third service, L4 selected the Interlibrary Loan Service by which library users 
are able to request books or articles from other libraries by submitting a request to the 
librarian. Users may retrieve materials personally at the library or have them mailed. 

3.2.6 Library 5 1 - (L5) 

3.2.6.1 Service 1 

For its first service, L5 chose the Automated Reference Service. In the library, the 
Automated Reference Service work stations are divided into two areas with LEXIS/NEXIS in 
one section and the others (ERIC, Psych Lit, Dow Jones, etc.) in another. L5 suggested we 
study both areas. 

3.2.6.2 Service 2 

L5's second selection was the traditional Reference Service. 

3.2.6.3 Service 3 

L5's third selection was the Patents Service, a patent depository where individuals may 
research an item/idea to determine whether it has already been patented. After searching at a 



1 For Wave I, this institution was studied separately from the other 4, with some modifications to the 
instrument. 
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work station the user gives the relevant patent number to the librarian who then gives the user 
a fiche with the patent. If the user already knows the patent number, the first step is 
eliminated. 

3.3 Quotas and Response Rates: Wave I 

3.3.1 Site Visits 

Early in the project, we conducted site visits to each library to obtain information about 
the feasibility of studying the services and to meet with librarians who would be "hosting" the 
interviewers. During those visits we observed users of the service, discussed the best 
positioning for the interviewers and determined the best hours for interviewing using an 
Impact Study Feasibility Report as a guide. (See Appendix F). At L5, the on-site project 
manager conducted the visits and reported her findings to us. 

Each interviewer was later given a set of Special Instructions for the particular 
service(s) (See Appendix F) based upon our findings during these visits. 

3.3.2 Establishing Quotas 

Prior to the onset of the study, we set approximate quotas for the number completed 
interviews for each service at each library. These quotas were based upon an overall goal of 
500 interviews: 100 per library evenly divided among the three services, (see Table 1). For 
budgeting purposes, we projected one completed interview per hour of interviewer time. 

3.3.3 Study Outcome; Response Rates 

Overall, in terms of interviews per hour, Wave I interviewing went slightly better 
than projected: 1.1 1 completed interviews per hour. Of all the in-person contacts, 26 percent 
resulted in completed interviews. About half (49%) of those contacted were ineligible 
(primarily because they were undergraduates) and about one-half of the remainder (23%) 
declined to be interviewed. Three percent of the interviews were terminated while in 
progress. The average number of completions per hour of interviewer time, for in-person 
interviews wa. c .95. 

The telephone interviews averaged 2.15 completed interviews per hour of interviewer 
time. Of all the telephone contacts (including answering machine responses, unavailable 
respondents and "no answers/busy signals), 24 percent resulted in completed interviews. 

Table 2 provides an overview of the completed interviews broken down by service and 
type of user. Table 3 gives an overview of the number of completions per hour by service. 
Table 4 shows the disposition of every contact made by the interviewers for in-person 
interviews, by service. Table 5 shows the disposition of every contact for telephone 
interviews, by service. The ;ables are followed by a discussion of the method used to study 
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each service as well as the details of the interviewing process. Overall, the low rate of 
eligible interviewees (51%) and the decision lo drop one service led to a total of 388 
completed interviews, 77% of the goal of 500. 



Table 1 
Wave I 
PROJECTED QUOTAS 




Name of Service 
LI 


Number of 
Interviews 


Reference Service 




Information Desk 


1 


Reference Desk - Library A 




Reference Desk - Library B 


1 A 
1U 


Reference Consultation 


0 


jvidicnais jueiivery oervice 




On-^itp recruitment 


17 


Online recruitment 

\S III lllv 1 Wi U1L1 1.1 VI 1 L 


17 


Ribliopranhic Instruction 


33 


JLZ 




ait ana Arcneoiogy - conecuon 




Diuiogy - ciecirunic rcierence 


J J 


Psychology - reference 


33 


L3 




Enhanced Online Catalog 




On-site recruitment 


17 


Online recruitment 


17 


Undergraduate Reserve 




Faculty 


i 


Students 


26 
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Table 1 (cont.) 
Wave I 
PROJECTED QUOTAS 




Science Libraries 




Carl UNCOVER 


17 


Document Delivery 


17 






L4 




Electronic Reference 


33 


Interlibrary Loan 


34 


Music and Media Center 




Faculty 


7 


On-site users 


26 






L5 




Automated Reference 


34 


Reference Service 


33 


Patents 


33 






TOTAL 


501 
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Table 2. Wave I: COMPLETED INTERVIEWS 



Service 


Grad Stud 


Faculty 


Undergrd 
Studnts 


Other 
Users 


Total 


LI 












Reference Service 












Information Desk 


7 








7 


Reference Desk-L A 


7 


2 


1 




10 


Reference Desk-L B 


10 








10 


Reference Consult 


6 








6 


Total 


30 


2 


1 




33 


Mat'ls. Deliv. Svc. 












Onsite recruitment 


25 


8 






33 


Online recruitment 




2 






2 


Total 


25 


10 






35 


TOTAL 


55 


12 


1 




68 


L2 












Art&Aroh-collection 


21 


6 


7 




34 


Biology-Elec. ref. 


15 


10 


8 




33 


Psychology-rcf. 


5 


5 


4 




14 


TOTAL 


41 


21 


19 




81 


L3 












Enhanced Online 
Catalog 












Onsite recruitment 


14 


1 






15 


Online recruitment 


12 


6 






18 


Total 


26 


7 






33 
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Table 2. (Continued) 



Undergrad. Reserve 












Faculty 




8 






8 


Students 






28 




28 


Total 




8 


28 




36 














Science Libraries 













Carl UNCO\ER 


3 


7 






1 n 
1U 


Document Delivery 


3 


i 

1 






4 


Total 


6 


o 

5 






14 


Hp* /""XT"" a T 

TOTAL 


32 


23 


2c 




o3 


L4 












Electronic Ref 


30 


3 






33 


Interlibrary Loan 


1 1 


1 
1 






1 0 


Music & Media Lntr. 












Faculty 




1 






/ 


On-site users 


15 


1 

J 






ZD 


lotal 




1 n 

1U 








TOTAL 


64 


14 






78 


L5* 












Automated Reference 


31 


3 






34 


Reference Service 


20 


2 


12 




34 


Patents 


2 




1 


7 


10 


Total 


S3 


5 


13 


7 


78 


TOTAL 


245 


75 


61 


7 


388 



*A different survey instrument was used lor L5. 
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Table 3. Wave I: COMPLETES PER HOUR* 



In-Person 



LI 



Information Desk 



Reference - Library A 



Library B 



Completes 



10 



10 



Hours 



10.5 



9.0 



12.0 



Compl/Hr 



.66 



1.1 



.83 



L2 



Art and Archaeology 



Biology 



Psychology 



L3 



34 



33 



14 



25 



33.25 



39.75 



1.36 



.99 



.35 



Online Catalog 



Reserve Students 



15 



28 



13.5 



24.25 



1.11 



1.15 



L4 



Electronic Reference 



Music & Media On-site 



L5 



33 



26 



19 



25 



1.73 



1.04 



Automated Reference 



Reference 



Patents 



Total 



Telephone 



TOTAL 



34 



34 



10 



288 



100 



388 



35.25 



26.25 



29.75 



302.50 



46.5 



349 



.96 



1.29 



.34 



.95 



2.15 



1.11 



Telephone ii 'erviews were conducted as a unit. Therefore, completes per hour for telephone 
interviews arc comprehensive. 
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3.4 Details of Survey Administration By Service 



3.4.1 LI REFERENCE SERVICE 

3.4.1.1 Information Desk 

To administer the survey, we intercepted users after they asked their questions at the 
reference desk. Interviewing for this service went moderately well. Although we were able 
to meet the quota, the overall number of completed interviews ("completes") per hour was 
.66, primarily because 71 percent of those contacted were (ineligible) undergraduates. 

3.4.1.2 Reference Desk 

At both libraries, we studied this service by intercepting users after they asked the 
reference librarian a question and before they left the reference area. 

3.4.1.2.1 Branch A 

Interviewing at Branch A generally went well. The quota was met with 1.1 completes 
per hour. Initially, however, the interviewer was stationed near the reference desk so that she 
could see who asked the query and then approach him/her afterward. This made one of the 
librarians somewhat uncomfortable and so we repositioned the interviewer farther away from 
the desk and closer to the exit. Interviewing continued successfully. 



3.4.1.2.2 Branch B 

The initial completion rate at this library was very low. Almost three-fourths of those 
contacted were undergraduates or had been interviewed previously. We chose to include 
undergraduates in order to meet the quota which we did with .83 completes per hour. 

3.4.1.3 Reference Consultation 

For this service, the reference librarian provided us with 12 names and telephone 
numbers of individuals who had used the service and we telephoned them from the APLab. 
We were successful in obtaining the six completes we had projected. 

3.4.2 LI; MATERIALS DELIVERY SERVICE (MPS ) 

We recruited respondents for this service using two methods: 

A) On-line-A request for volunteers was placed on the campus- 
wide INFO Service menu (see Appendix F). Upon selecting 
"Libraries", then "PN9" for public news, the menu choice 
"Materials Delivery Service Survey" was offered. Respondents 
were invited to respond by e-mail or telephone the APLab to 
volunteer. 

B) On-site— The librarians attached a form requesting that faculty 
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and graduate students consent to participate (see Appendix F) to 
every MDS request form distributed at libraries on the three 
campuses. The forms were then submitted to circulation desk 
staff and forwarded to us. 
The survey was then administered by telephone from the APLab. 

The on-line recruitment produced only two volunteers (both leading to completed 
interviews). To compensate for this low response rate, we increased the quota for the on-site 
interviews. From 86 completed consent forms, we obtained 33 completed interviews. 



3.4.3 LI: BIBLIOGRAPHIC INSTRUCTION 



To compile the sample, LI provided us with a list of faculty whose students had 
pai ucipated in BI. From that list, we requested they choose one faculty member for each of 
the courses we selected: 



English 102 
Biology 100 

Social Responses to Environmental Problems OR Sociology of Gender 



and telephone them to request participation. 



At that point, LI determined the data collection for this service was not feasible. Due 
to the late date of this decision, library LI elected to waive their third selection. 



3.4.4 L2 : ART AND ARCHAEOLOGY - THE COLLECTION 



In this library users were intercepted and interviewed as they left the library. Initially, 
we had difficultly obtaining the requisite number of completes per hour. Although only 45 
percent of those contacted were ineligible, slightly more than a fourth (29%) refused to be 
interviewed. As a result, we permitted the interviewers to accept undergraduate users. 
Further, we discovered that because it was a non-circulating collection, students were inclined 
to spend a lot of time in the library and it was important to interview during the hours when 
patrons left for meals. Ultimately, we met the quota with 1.36 completes per hour. 

3.4.5 L2: BIOLOGY - ELECTRONIC REFERENCE 

In this library, respondents were intercepted and interviewed as they left the work 
stations. Aside from having to open up the sample to undergraduates due to the low traffic, 
interviewing for this service went smoothly; the quota was met with a completion rate of .99 
per hour. Only 35 percent of those contacted were ineligible and only 5 percent refused to 
participate. 



3.4.6 L2: PSYCHOLOGY - REFERENCE 



We began studying this service by intercepting and screening users after they asked the 
reference librarian a question. However, our completion rate was so low we decided to add 
users of the electronic services and to add undergraduate users. Neither tactic was as helpful 
as anticipated. After consulting with tl.°, librarian, we changed the interviewer and the 
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"interviewing days but that did not increase the completion rate either. We were able to 
complete only 14 of the anticipated 33 at a completion rate of .35 per hour. An analysis of 
the contact dispositions revealed that only 33 contacts were made in total, leading us to 
conclude that the library did not have enough traffic for us to meet our quota. The librarian 
confirmed our observation. She noted that most users were already familiar with the sources 
they required and did not need to ask the reference librarian questions or search the electronic 
reference services. 

3.4.7 L3: ENHANCED ONI INK CATALOG 

Interviews for this service were initiated in two ways: 

A) On-site— Users were intercepted and interviewed as they left the enhanced online 

catalog work stations at one of the university libraries. 

B) Online—Upon accessing the enhanced online catalog from a remote location users 

were provided with the following menu option: 

#9: How are we doing?--Evaluation of (Service Name). 

After choosing this option, they were invited to participate and to provide 
information so we could contact them (see Appendix F). 

Responses were then electronically forwarded to a project staff member and the 
survey was administered by telephone from the APLab. 

3.4.7.1 On-site 

Thi. service went exceptionally smoothly. The completion rate was 1.11 per hour. 
(We were two interviews short of the quota because of an interviewer illness.) 

3.4.7.2 Online 

We had a total of 41 online offers to be interviewed; 29 of whom were graduate 
students or faculty. The remainder were undergraduates. Candidates were telephoned from 
the APLab. Interviewing went well and the quota was met. 



3.4.8 L3: UNDERGRADUATE RESERVE 

3.4.8.1 Faculty 

To study faculty, L3 provided us with 20 names and telephone numbers of instructors 
who had placed materials on reserve. We telephoned them from the APLab and were able to 
exceed our quota. Because of the nature of this service, there was some non-uniformity in 
how the respondents interpreted the questionnaire. Sometimes faculty answered the questions 
in terms of the benefits they received and other times in terms of the benefits their students 
received. 
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3.4.8.2 Students 



To study the students, we intercepted them and administered the questionnaire after 
they returned the borrowed items. Again, the questionnaire did not always make sense to 
these users because of the nature of the service, particularly questions about their length of 
use. Nevertheless, we met our anticipated quota with a completion rate of 1.15 per hour. 

3.4.9 L3: SCIENCE LIBRARIES 

3.4.9.1 Document Delivery 

Volunteers for this study were recruited in two ways: 

A) In-person pick-up— For patrons who picked up material at the 
library, our request form (see Appendix F) was attached to each 
document when it was retrieved, then returned to the librarian 
upon completion. The forms were then forwarded to us. 

B) Mailed items— For those documents delivered by mail, a return 
postage-paid postcard (see Appendix F) was included with the 
item so the user could mail it to us directly. 

We administered the survey by telephone for both types of candidates. 

For both methods of recruitment, we got so few volunteers from the Biology Library that, 
after discussing it with the librarians, we decided to expand this part of the study to the 
Psychology, Engineering and Math/Science Libraries. Despite this expansion, we received 
only eight returns from the request forms that were distributed at pick-up points. Out of 100 
postcards given to the Biology Library initially, we only got back three. Therefore, we were 
only able to obtain 4 completes out of the 17 we had projected. 

3.4.10 L3: Carl UNCOVER 

L3 provided us with the names and telephone numbers of faculty numbers who had 
participated in the program and the survey was administered by telephone. From the 1 1 
names, we were able to obtain 10 completed interviews. 

3.4.11 L4: ELECTRONIC REFERENCE 

To study the Electronic Reference Resources at L4, we intercepted and interviewed 
users as they left the work stations. To ensure that we obtained users on all three floors, the 
interviewers' locations were varied from shift to shift. Interviewing for this service went very 
well; the quota was met with a completion rate of 1.73 per hour. 
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3.4.12 L3: MUSIC AND MEDIA CENTER 



3.4.12.1 On-site Users 

To study on-siti users at the Center, we intercepted them and administered the survey 
after they finished using the service/returned the borrowed item. The questionnaire was 
slightly awkward for this service but generally it went well. We met the quota with a 
completion rate of 1.04 completes per hour. We discovered that in order to "catch" those 
users who came to watch films after lunch, it was important to schedule the interviewer for 
late afternoon when the films ended. 

3.4.12.2 Faculty 

For the faculty users, L4 provided us with a list of 9 faculty members who had placed 
materials on reserve and agreed to be contacted. We administered the survey by telephone 
and obtained the 7 anticipated completes. 

3.4.13 L3: INTERLIBRARY LOAN 

We recruited volunteers for this service in two ways: 

A) In-person~Our request form was handed to users when they 
retrieved their materials and then returned to library staff and 
forwarded to us. 

B) By mail—For materials that were mailed to users, a postage-paid 
postcard requesting participation in the study was enclosed with 
the item so the user could mail it directly to us. 

We administered this survey by telephone to volunteers of both types. 

Only 17 consent forms (plus an additional seven after the closing deadline) were returned 
to us and out of 100 postcards sent out, only 10 were returned. A total of 10 interviews were 
completed. 

3.4.14 L5: AUTOMATED REFERENCE 

To study this service, we established quotas (7 for LEXIS/NEXIS and 27 for the other 
services) so that users of both sections were intercepted and administered the survey. 
Interviewing for this service went well. The quota of 34 interviews was met with a 
completion rate of .96 per hour. 

3.4.15 L5: REFERENCE SERVICE 

As with other reference services, users were intercepted and the survey administered after 
they asked the reference librarian a question and before they left the reference area. 
Interviewing for this service went very well; the quota was met with a completion rate of 
1.29 per hour. 
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3.4.16 L5: PATENT SERVICE 



To study this service, we intercepted users and administered the survey as they left the 
patents area. Because of the low completion rate, part of the way through the interviewing 
we opened up the service to both undergraduates as well as non-university affiliated users as 
the on-site manager suggested. However, this did not alleviate the problem. Only 18 
contacts were made, indicating that there were not enough users during the interviewing 
period. We were able to complete only 10 of the anticipated 33 completes at a completion 
rate of .34 per hour. 

3.5 THE INTERVIEWERS 



3.5.1 Recruiting Interviewers 



To hire the interviewing staff for LI, L2, L3, and L4 we posted recruitment notices 
(see Appendix F) at the Rutgers School of Communication, Information and Library Studies 
(which houses the Journalism, Communication, and Library and Information Science 
programs), and at the on-campus student employment offices on the three Rutgers campuses. 
In addition, we asked the Journalism Department chairperson to recommend outstanding 
undergraduate journalism students. We also requested that library science and journalism 
faculty announce the position in their classes. 

Simultaneously, we investigated the possibility of hiring a professional market research 
field service to conduct the interviews. We found this would forfeit some quality and cost 
control and chose to proceed with the original plan. 

The recruitment notice instructed interested applicants to contact the Project. Manager. 
When they did, she gave them a cover letter and application form (see Appendix F). 

Of the 21 applications returned, she scheduled interviews with 16 applicants. (Some 
were no longer interested, others' applications were received after the closing deadline, others 
were not available during the necessary times). 

3.5.2 Interviewing the Applicants 

During the interview, the Project Manager asked the applicants to elaborate on: 

Any interviewing experience (if relevant) 

Other work experience 

Academic status 

Availability 

Preference for location 

Preference for phone vs. in-person interviewing 

Maximum and minimum number of hours they were willing to work 

Knowledge of libraries 

Personal interests (to get a sense of personality) 
Availability of a home telephone 
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Where appropriate, the relevant information was recorded on the application. 



The Project Manager also noted the applicant's degree of eye contact, friendliness, 
professionalism, assertiveness and overall presentation, and recorded any outstanding traits on 
the application form. 

After interviewing the applicants, the Project Marager selected those who seemed 
most promising and compared their availability with the project needs. Based on the original 
estimate of about one completed interview per hour and a work schedule of approximately 
10-15 hours per week, she hired eight interviewers and one substitute. Each interviewer was 
sent a letter offering a part-time position and outlining the job requirements, with a request 
that they return a signed copy (see Appendix F). 

At L5, the on-site Project Manager recruited, interviewed and hired 4 interviewers and 
one substitute using the same procedures as the Project Manager at Rutgers. 

Each interviewer was assigned a unique interviewer number for record-keeping 
purposes. 

3.5.3 Training 

L1.L2.L3.L4 (Training at Rutgers/SCILS) 

All interviewers (including the substitute) were required to attend a two-hour, paid 
training session given by the Project Manager at Rutgers. The session was held on the 
Saturday morning prior to the first week of interviewing. On the application form all the 
interviewers had indicated they were available at that time. 

Before the session began, each interviewer (excluding the substitute) was given a 
packet of supplies along with the following: 



Questionnaires (for their assigned service(s)) 
Questionnaire Instructions 
Tally sheets 
A time sheet 

An Interviewing Assignment/Schedule 
A Special Instruction Sheet 
An Interviewer Training Manual 
A supply sheet 



Attendees were encouraged to review the materials while waiting for the session to begin. 



3.5.3.1 The agenda for the training session was as follows: 



3.5.3.1.1 Welcome: 5 minutes 

The Project Manager thanked the interviewers for attending the session then described 
the study's purpose and the interviewers' role in the study. She then told them the two most 
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important points to be covered during the session: 

1) To probe open-ended questions. 

2) To be sure all interviews are taped and audible. 
She then invited them to ask questions freely during the session. 

3.5.3.1.2 Questionnaire Familiarization - 10 minutes 

The interviewers were then requested to pair up and, with one person acting as the 
respondent and the other as the interviewer, read half the questionnaire to their partner, then, 
midway through, switch roles. This exercise was conducted to give the attendees first hand 
experience with the questionnaire, so that the remainder of the session would be more 
meaningful them. Also, because some of the interviewers would be dealing with more 
than one service (and therefore, more than one version of the questionnaire) during the course 
of the study, for the entire training session, each interviewer used a questionnaire from his or 
her own packet. All were different. 

3.5.3.1.3 Interviewing- 45 minutes 

For this portion of the training session, the instructor went over the most important 
concepts from the Interviewer Training Manual (see Appendix E) citing examples where 
appropriate and emphasizing the following: Interviewing is one of the most important parts 
of the study; without it there is no data. There are certain procedures an interviewer can 
follow to guarantee we get high quality, accurate and valid information. They are: 

- Develop rapport. 

- Read the questionnaire as written. 

- Know the different types of questions. 

- Probe for the most specific response. 

- Know how to handle a "don't know/refusal". 

- Edit the questionnaire before handing it in. 

During the discussion about probing, the Project Manager illustrated the technique by 
questioning some of the interviewers (i.e. Why did they choose their major?, Why did they 
attend the training session?, etc.) and probing their responses. She then had one of the 
interviewers question another and critiqued the probing skills. 

Interviewers were instructed to tape one interview per side of the tape, to write the 
interview number on the tape, and to use pencil when interviewing. 

At the conclusion of this portion of the session, the instructor asked the attendees to 
return to page 1 of the Training Manual, and had each attendee read one of the Summary 
Points. They were also told to read the Training Manual on their own before their first shift. 

3.5.3.1.4 Break - 5 minutes 



3.5.3.1.5 Questionnaire/Instructions/Tally Sheet - 30 minutes 

During this portion of the session, each interviewer read a question from the 
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Questionnaire and the Project Manager reviewed the' corresponding instructions from the 
Questionnaire Instructions. She also reviewed the procedure for completing the Tally Sheet 
and for numbering the questionnaires. 

3.5.3.1.6 General Rules/Information - 20 minutes 

At this point, the instructor asked each interviewer to look over his or her Special 
Instructions Sheet and presented miscellaneous general instructions as follows: 

1. Keep the cassette recorder's VOR button in the "off" position and the sensitivity on 

"high". 

2. Be sure the microphone is near the interviewer's mouth. 

3. Introduce yourself to the contact person upon arriving at the interviewing site. That 

person will tell you where to position yourself to intercept interviewers. 

4. If a question arises at the interviewing site, call the office collect and ask the 

question rather than possibly proceed erroneously. 

5. Wear your name tag at all times. 

6. Dress neatly/professionally (preferably no jeans) as you are representing Rutgers 

University. 

7. Try to obtain as many completes at each shift as possible. 

8. Be aware of your screening requirements. 

9. Bring your "compMwtes" and Tally Sheets to the APLab the day after your shift 

unless other arrangements have been made, 

10. Call the day before to find out if you need to come in for your phone shift. 

11. To be reimbursed for travel you must present your receipts. 

12. How to fill out the Time Sheet. 

13. If you require directions to the interviewing site, please see the Project Manager 

after the training session. 

14. To determine how many tapes you need, take your quota and divide it by 2 then 

add 5. (One interview per side plus 5 extra.) 

15. The record keeping for telephone interviewing differs slightly from in-person 

interviewing. You will be instructed on this at your first phone shift. 

16. Practice interviewing with a roommate, spouse, friend, etc. before going to your 

first interview. 

In addition, interviewers were given the Project Manager's office hours. They were 
also advised that their Time Sheets would be submitted to the Accounting Office for payment 
after all equipment was returned. 

3.5.3.1.7 Practice Interviewing ■ 10 minutes 

The Project Manager then asked the interviewers to pair up and conduct one taped 
interview each with their partner. 

3.5.3.1.8 Closing 

Interviewers were thanked for attending and asked to take their tapes and record how 
many they took on the sign-up sheet. They were also given an opportunity to sharpen their 
pencils. 
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3.5.3.2 Training at Library L5 



For L5, the training session was conducted on-site by one of the Project Directors who 
used the same training session materials and followed the same outline used at Rutgers. 

3.5.4 Reviewing the Work of Interviewers 

After each interviewer's first shift, the Project Manager listened to his/her tapes and 
reviewed the written questionnaires to check the quality of the interviewing. Afterward, the 
Project Manager discussed her observations with the interviewer or in the case of L5, with the 
on-site Manager. If necessary, the Project Manager repeated the procedure for the following 
shift, until any difficulties were resolved. Throughout the remainder of the interviewing 
period, the Project Manager randomly chose a few interviews and listened to them for quality 
control. 
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3.6 THE RESEARCH INSTRUMENTS 



3.6.1 Focus Groups 

Prior to the survey waves we conducted a preliminary phase which consisted of two focus 
groups at two of the libraries. For the LI group, a senior library administrator invited 13 
university faculty members who use the library regularly. Of the 13, 11 attended the group. 
A variety of disciplines were represented, including religion, sociology, mathematics, art 
history, history, human ecology, chemistry, economics, english, and the alcohol studies center. 
At L2, a senior library administrator recruited 14 graduate student users from the three 
libraries studied there, 13 of whom attended the group. The students, three from the biology 
department, four from psychology and six from art history, were referred to the administrator 
by the librarians at the individual libraries. The sessions, both of which were tape recorded, 
were held from 12:00-1:30 p.m. and lunch was served. 

The groups were conducted by a trained focus group moderator using a Moderator Guide 
(see Appendix G) which addressed all of the issues in the preliminary draft of the survey 
instrument including: 

1. Library Usage 

2. Benefits of Library Usage 

3. Measurement of Library Usage 

Tape recordings of both groups were transcribed for further review. A preliminary 
analysis of the groups confirmed our suspicion that users describe the benefit, or lack of 
benefit, of library services in terms of time and money. In addition, they allude to other less 
economically tractable measures such as their perceptions of the institution as a whole, 
external perceptions of the institution, and their ability tc "do what they do" as scholars. 

We proceeded with questions to assess the benefits of library services in time, money and 
convenience in the quantitative portion of the study. Further, as a direct result of one faculty 
member's comment that she does not give "points" to the library for having what she wants 
because she expects that it will, yet deducts them when it does not (similar to the way she 
grades an exam), we added a "grading" question about the users' experience with the service. 

3.6.2 The Questionnaire 

3.6.2.1 LI. L2, L3. L4 (Version 1.1) 

In developing the survey instrument for the quantitative portion of the study, we used the 
following topic areas as our guide: 

- The user's learning curve for the service (i.e. knowledge, experience, skill level, etc.) 

- The presentation of the user's information problem. 

- The effectiveness of the service (i.e. value --time and dollar, benefit, impact, convenience, 
satisfaction, success, etc.). 
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*- The affective responses to the service (satisfaction, comfort, difficulty, frustration, etc.). 

After numerous modifications, we arrived at a master version for pretesting. As a 
preliminary pre-test, we conducted two telephone interviews with personal acquaintances who 
had recently used a library. We then obtained permission from one of the participating 
libraries and conducted three interviews with actual library users. 

During the pre-test, we corrected some typographical errors and omissions. We 
developed two final master versions of the questionnaire: one for telephone interviews and 
one for in-person interviews (see Appendix A). On the master versions, we left easily 
identifiable symbols for word-processor insertion of the library and service names. We also 
added coding columns, detailing the column in which the response to each question would be 
recorded for computer input. Once the 17 basic versions were created, we modified each one 
as appropriate for the particular sen-ice (some questions were omitted, some of the wording 
was changed slightly, etc.) and circled the corresponding numerical code for the Library, the 
Service, and the Version. (In some cases we studied more than one aspect of the same 
service.) 

On the questionnaires, all interviewer instructions were printed in capital letters and 
critical instructions were emphasized with bold print. Information to be read by the 
interviewer was printed in normal type. For closed-ended questions (with predesignated 
responses) the interviewer needed only to circle the number corresponding to the response 
given. For actual numerical responses, interviewers were requested to use whole numbers 
only. Because the interviews were taped, interviewers did not need to record resMwnses to 
open-ended questions. 

A set of detailed Questionnaire Instructions, with detailed instructions for each 
question (see Appendix H) was prepared. 

3.6.2.2 Instrument at Library L5 (Version 1.2) 

Because L5 was recruited after the other libraries, we conducted the Wave I interviews 
for L5 later than the others. We used the delay to perform preliminary analysis, and to 
modify the questionnaire and experiment with new questions based upon what we learned 
from this analysis. As a result, the order of the questions was changed, some of the original 
questions were deleted, and new open-ended questions were added (see Appendix A). The 
new questionnaire was pretested at one of the participating libraries and again we created a 
master version, inserted the library and service names where appropriate and pre-marked the 
Library, Service and Version. Questionnaire Instructions were also prepared. 
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3.7 PROJECT MANAGEMENT 



3.7.1 Human Subjects Protection 

Because we were conducting research with human subjects, we filed a "Request for 
Exemption" with the Rutgers University Institutional Review Board for the Protection of 
Human Subjects in Research. We were granted an exemption from having our respondents 
complete a consent form, on the grounds that our questionnaire was anonymous and fell 
within Rutgers University guidelines for exempt research. 

3.7.2 Scheduling Interviewers 

We decided that interviewing would take place only during weekdays because we 
would be more likely to find faculty users and less like to encounter non-university affiliated 
users. 

3.7.2.1 In-Person Interviews 

To schedule the on-site interviewers, the Project Manager converted the quota for each 
service to interviewer hours per week and, where appropriate, added travel time. She then 
matched a list with of interviewers' preferences and availability with the information from 
the Impact Study Feasibility Report. With primary emphasis on covering each service during 
the most productive times, she developed a schedule for each service. In all but one case it 
was possible to assign interviewers to cover the best time slots. For the one exception, the 
interviewer was scheduled at a less busy time. 



3.7.2.2 Telephone Interviewe rs 



To schedule the telephone interviews, the Project Manager divided the day into three 
shifts: 9-12:00, 1-5:00, and 6-9:00. She then made a list of the available hours remaining for 
the interviewers after their in-person schedules were established, and scheduled one or two 
people for each time slot. 

Once the schedules were set (about a week prior to the onset of the interviewing), the 
Project Manger telephoned each interviewer with the schedule and location. In addition, at 
the training session (see below), each interviewer was given a written Interviewing 
Assignment/Schedule. (See Appendix F). The interviewers were advised th»t due to the 
uncertain outcome of the interviewing the in-person schedule was for the first week only and 
subject to change as the study developed. Also, they were to telephone the APLab prior to 
their scheduled phone shift to determine if there was work. 

In addition to providing the interviewers with their schedules, the Project Manager 
advised the librarians at each library as to the interviewer's schedule, again reminding them 
that these were tentative until the conclusion of week 1. Throughout the study the Project 
Manager remained in close contact with the participating librarians by phone or electronic 
mail. 
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3.7.2.3 Interviewer Numbers 



Each interviewer was assigned a unique interviewer number which was recorded on 
every questionnaire he/she completed. 

3.7.2.4 Equipment/Supplies 

To tape the in-person interviews we purchased Sony portable cassette recorders 
(Model TCM-S64V) for the interviewers. They cost approximately $30 each. For the 
telephone interviews a microphone was attached to the telephone ear-piece and to a desk-top 
cassette recorder. 

In addition, we gave each interviewer the following supplies: 

Cassette tapes 

4 AA batteries (2 for the recorder and 2 extra) 
A clip board 
A name tag 
Pencils 

A portable pencil sharpener 

3.7.2.5 Numbering the Questionnaires 

To number the in-person interviews, we instructed the interviewers to begin with their 
interviewer number and then, beginning with 01, continue on consecutively. Thus, 
interviewer 6 coded interviews as 601, 602. ..etc. In cases where the interviewer was working 
on more than one service, we set a starting number for each service allowing for the quota. 

For the telephone interviews, which were conducted from the APLab by a few 
interviewers, all the interviews were numbered beginning with "8" or "9" (depending upon the 
telephone used), regardless of the service. Mw For L5, because there were no phone 
interviews, all interviewers were instructed to begin numbering the completed interviews with 
001 and continue on consecutively with the understanding that when processed the number 
would be preceded by a unique interviewer number. In retrospect, we believe that a fixed 
numbering scheme should oe imposed at the outset, and maintained thorughout the project. 

3.7.2.6 Tally Sheets 

For the in-person interviewing every interviewer was required to keep a Tally Sheet 
for every shift. (See Appendix F). On the Tally Sheet, the interviewer made a tick mark in 
the appropriate box for every potential respondent approached using the following guidelines: 

Ineligible respondent: an individual who does not qualify for the study based upon the 
responses to the questions on page 1 of the questionnaire. 

Incomplete: an interview terminated by the respondent before completion. (The interviewer 
actually wrote on the questionnaire from Q.l on.) 

Interviewer terminate: an interview terminated by the interviewer because of a language 
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'problem, hearing problem, etc. 

Refusal: a potential respondent who refused to be interviewed after being approached. 

Complete: an interview in which the respondent's answers have been recorded on the 
questionnaire and on the tape recorder and the interview number recorded on the tape. 

Once the interviewing began, the interviewers were asked to bring their completed in- 
person interviews and Tally Sheets to the APLab after each shift. Upon receiving them, the 
Project Manager recorded the number of completed interviews and hours worked to determine 
the average number of completes per hour for each service. 

At L5, we asked the interviewers to record the exact hours worked on the Tally Sheet 
so we could calculate completes per hour more efficiently. 

3.7.2.7 Call Disposition Sheets 

For phone interviews, the interviewers were given a set of Telephone Interviewing 
Instructions (see Appendix H) and were required to record the outcome of every phone call 
on the Call Disposition Sheet (see Appendix F) following the same 
guidelines as above. 

3.7.2.8 Editing 

Throughout the interviewing period, upon receiving the completed questionnaires, the 
Project Manager edited each one. This included: 

-Checking to see the questionnaire was filled out completely and accurately. 
-Drawing a slash through any skipped questions (to help increase the keypuncher's 
speed). 

-Coding the Area Study/Department question. 

3.7.2.9 Keypunching/Transcribing 

Once the interviews were completed, the closed-ended responses from the 
questionnaires were keypunched and the open-ended questions transcribed. After obtaining 
cost estimates and calling references, we hired outside suppliers for both tasks. 

3. 7.2. 9. 1 Keypunching 

Along with the edited questionnaires, we provided the keypuncher with a list of: the 
questions, our preferred variable name, the corresponding coding columns and the valid 
responses for each one. In addition, we asked that he skip over the open-ended questions, 
punch for "no responses" and punch each questionnaire twice to verify the entries. 

We received a floppy disk with the data as well as a hard copy, both in a lotus 123 
spreadsheet format. Upon receiving it, we randomly chose 10 questionnaires and checked 
that each was punched without any errors. We requested data in Lotus- 1-2-3 format, for easy 
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transfer into SPSS data file format. 

3. 7.2. 9.2 Transcribing 

After an initial disappointing experience with a transcriber who did not transcribe the 
responses verbatim, we switched to another and had her do five interviews which we checked 
carefully. Once satisfied, we gave her a few more and checked them. Satisfied again, we 
gave her the remainder and upon receiving the transcripts, we randomly selected a few and 
checked them carefully. 

The transcriber provided us with a hard copy of each transcript (See the Report, Part 
1 , for an example) as well as a floppy disk with all of the transcriptions in Word Perfect 
format. In retrospect, more detailed definition of the format would have simplfiied machine 
processing of those transcripts. 

3.7.2.10 'Labor Costs 

The total labor cost for administering, keypunching and transcribing the Wave I 



questionnaires are outlined below: 



Task 


Quantity 


Rate 


Cost 


Interviewers 
(Training and 
Interviewing) 


442.25 hrs. 


$10/hr. 


$ 4,422.50 


Keypunching 


388 


.80/interview 


$ 310.40 


Transcribing 


388 


$8.00/interv. 


$ 3,104.00 


TOTAL 






$ 7,836.90 



The average cost per interview was: $20.20. Travel costs have been omitted. The cost of 
administering and supervising the work of the interviewers is difficult to separate from the 
work of developing manuals, instruments and forms. At a minimum it is 40 hours for 
recruitment and training, 120 hours supervising data collection, and 40 hours dealing with 
keypunch service and transcription service. During Wave 1.1, a total of 11 services were 
studied. We estimate that 30% of the administrative costs are fixed costs and the remaining 
70% are variable. This results in an estimated administrative cost of 

(0.3) x (200 hrs.) + N/1 1 x (0.7) x (200 hrs.) 

= 60 hrs. + 12.7 x N hrs. 

where N is the number of services. 
Because Wave 1 was conducted in two parts the total administrative costs are estimated at 

60 + 11/Q x 12.7 

+ 60 + 3 x 12.7 = 298 hours 
The administrative cost is given in hours rather than dollars because the work was divided 
between project manager and principal investigator. The project manager should be an 
i dividual with at least 18 months experience in survey research. 



APLAB-B%D\CLR\FINAL.REV\PROCESS.51 Copyright (c) 1995 March 21, 1995 P. 36 



120 



4 DELAYED IMPACT ASSESSMENT: WAVE 2 



The primary goal for Wave 2 was to explore the long-term impact of the services 
used. As such, the study consisted of either an initial in-person interview which we called an 
"Audition" followed by a telephone "Callback" two weeks later OR a "Phone" interview two 
weeks after the individual used the service. The phone interview was used for services that 
do not bring the user to the library. 

4.1 Schedule of Wave 2 

Again, we established an approximate schedule (taking into consideration the Spring 
breaks at all institutions) as follows: 

1/3-3/4/94 Recruit Interviewers/Develop Questionnaires 
3/7-3/11/94 Pretest Questionnaire 
3/21-3/25/94 Train Interviewers 
3/28-4/8/94 In-person Interviewing 
4/11-4/29/94 Telephone Interviewing 

Overall, we were able to keep to the schedule. 

Data collection for Wave I was managed as in Wave II. 



4.2 THE SERVICES 



For Wave II, we studied most of the same services as in Wave I. Some were 
eliminated based upon our experiences in the first wave. A discussion of the services 
follows: 



4.2.1 Library LI 

At LI we studied the same 2 services as in Wave II; once again, LI elected not to 
select a third service. 

4.2.2 Library L2 

At L2 we also studied all of the same services. However, at the Psychology library, 
we included users of the Electronic Reference Services from the onset, based upon our 
experience in Wave I. 

4 .2.3 Library L3 

At L3 we studied the Enhanced Online Catalog and the Science Document Delivery 
service again. We eliminated the undergraduate Reserve service based on our fii dings, from 
Wave I, that it was not particularly well suited to the nature of the study. We also found that 
we had exhausted the list of Carl UNCOVER pilot project names during Wave I and, 
therefore, we were unable to study that service again. 
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4.2.4 Library L4 



At L4 we studied the Electronic Reference and the Interlibrary Loan services again. 
We did not include the Music and Media center because, like the undergraduate Reserve 
service at L3, it was not suited to our methodology or questionnaire. 

4.2-5 Library L5 

At L5 we studied all of the services that we studied in Wave I. 

4.3 Recontacting Librarians 

Prior to the start of Wave 2, we contacted all of the libraries where we had difficulties 
(e.g. in meeting quotas) during Wave I, to determine if there was anything we could do to 
improve our data collection methods for this Wave. In response, the L2 Psychology librarian 
suggested we position the interviewer at the library exit rather than inside the library near the 
reference desk so as not to intimidate the librarian. 

4.4 Establishing Quotas 

Prior to the start of Wave 2 we again set approximate quotas of completed interviews 
for each service at each library. (See Table 6). Our plan was to conduct 50 initial 
("Audition") or full telephone interviews at each library (34 at LI where we only studied two 
services) and then recontact ("Callback") as many of the Audition respondents as possible two 
weeks later. For the Telephone interviews we aimed to wait two weeks after the candidates 
had volunteered so that they had time to actually use the materials they had obtained. 

4.5 Study Outcome: Response Rates 

Overall, Wave 2 interviewing went better than Wave 1: 1.38 completed interviews per 
hour. The average number of completes per hour for in-person ("Audition") interviews was 
also slightly higher at .99 per hour. The Callback and Telephone interviews, combined, 
averaged ?.28 completes per hour. Of all the in-person Audition contacts, about 30 percent 
resulted in completed interviews. Fifty-six percent were ineligible and 12 percent refused to 
be interviewed. Fewer than 5 percent of the interviews were terminated while in progress. 
Of all the Callback contacts, 17 percent resulted in completed interviews; of all the Phone 
contacts, 19 percent resulted in completed interviews. Overall, 69 percent of the auditions 
resulted in a successful Callback. 

Table 7 provides an overview of the completed interviews broken down by service and 
type of interview. Table 8 gives an overview of the number of completes per hour by 
service. Table 9 shows the disposition of every contact made by the interviewers for in- 
person interviews, by service. Tables 10 and 1 1 show the disposition of every contact for 
Callback and Telephone interviews by service. Table 12 presents the percentage of Auditions 
that resulted in a successful Callback, by service. The tables are followed by a discussion of 
the outcome for each service. 
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Table 6: Wave II Projected Quotas 


Name of service 


Number of 


LI 




Reference service 




Information desk 


4 


Reference desk-Branch A 


4 


Reference desk-Branch B 


4 


Reference consultation 


4 


TOTAL 


16 


Materials delivery service 


17 


TOTAL* 


33 


L2 




Art & Archaeologv-collection 


17 


Biology-electronic reference 


17 


Psychology-reference 


16 


TOTAL 


50 


L3 




Enhanced online catalog 




On-site recruitment 


13 


Online recruitment 


13 


TOTAL 


26 


Science document delivery 


24 


TOTAL 


50 


L4 




Electronic reference 


25 


Interlibrarv loan 


25 


TOTAL 


50 


L5 




Automated reference 


17 


Reference service 


17 


Patents 


16 


TOTAL 


50 








_JT3 _ 
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Table 7: Wave 2: Completed Interviews 




A nHitinn 

AUUl LI VJil 


Callhack 




T 1 

JL> 1 








Reference Service 








Information Desk 


4 


3 




Reference Desk-Library A 


4 


3 


—i 


Reference Desk-Librarv B 


4 


2 




Reference Consultation 






l 


Materials Delivery Service 




L O 


T 0 








Art & Archaeologv-Collection 


1 H 
1 / 


i s 




Biologv-Electronic Reference 


1 H 
1 / 


s 

o 




Psvcholoav-Reference 


10 


1 1 




L3 








Enhanced Online Catalog 








Ons ite 


13 


Q 




Online 






1 9 


Science Document Delivery 






o 


L4 








Electronic Reference 


25 


i <: 
1 j 




Interlibrarv Loan 






o 
z 


L5 


17 


14 




Automated Reference 


17 


13 




Reference Service 


10 


7 




Patents 


10 


7 




Tnfjl 


ISA 


1D7 


39 
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Table 8. Wave 2: Completes Per Hour 



Complete 



Hours 



Completes 
Per Hour 



Auditions 



LI 



Reference Service 



Information Desk 



9.25 



Reference Desk-Branch A 



Reference Desk-Branch B 



L2 



Art&Archaeology-Collection 



17 



7.75 



Biology-Electronic Reference 



17 



18.5 



Psychology-Reference 



16 



13.5 



L3 



Enhanced Online Catalog 



13 



10 



L4 



Electronic Reference 



25 



18.5 



L5 



Automated Reference 



Reference Service 



17 



17 



16 



17.5 



Patents 



10 



26.75 



TOTAL 



1 44 



144.75 



CALL BACKS and PHONE* 



139 



61 



TOTAL 



283 



*Callbacks and phone interviews were conducted as a unit 



205.75 
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:RIC 



APLAB-8%D\CI.R\FINALREV\PROCESS.51 



Copyright (c) 1995 



125 



March 21, 1995 P. 41 



01 



-c 
p 



10 



o 

o 



10 

CO 

co 

UJ 

o 
o 

C£ 
a. 

S 

cc 
-J 

a: 
~j 
o 

Q 

5? 
CO 

CD 

a. 




aS 
U 
O 



# 
O 
O 



# 
O 
o 



o 
o 



# 
O 
O 



O 
O 



# 
O 
O 



o 
o 



i 

o 
o 



01 

OS 



00 



CV? 

r-1 



0) 

A! 



w 
CQ 



o 



Pi 



■C 



111 
> 

s 



.a 
ns 



& 
o 
u 
a 



u 



u 
as 



111 
Q 



U 
OS 



111 

o 

111 
u 

d 

111 
u 
111 



o 

o 
111 

as 



< 



111 



111 



0 
O 



d 

o 



o 



(J 



a 

o 

n 
ai 
o 
a 

as 

■g 



d 

111 

111 
y-i 
m 



d 
<u 
u 
m 
y-i 
ai 

OS 

•0 
o 
u 



3 



m 
u 

-H 

> 

w 
ai 



111 
a: 



< 

o. 



O 
cc 
a. 
S 
Uj 
CC 
-J 

cc 
— i 

CJ 
Q 

CQ 
CQ 
— I 

a. 



CO 



9 

ERIC 



Table 12: Wave 2: Percentage of Auditions Resulting in Callbacks 
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4.5.1 LI: Reference Service 



4.5.1.1 Information Desk 

For this service, our number of "completes per hour" was only .43. We elected not to 
include undergraduates in order to be consistent with Wave I. Of the u>ur completed 
Audition interviews, we were able to obtain three successful Callback interviews. 

4.5.2 LI: Reference Desk 

4.5.2.1 Branch A At this branch we obtained one completed interview per hour. Of the four 
completed interviews we were able to obtain three completed Callback interviews. 

4.5.2.2 Branch B Based on our experience during Wave I we included undergraduates in the 
sample from the onset. The completion rate here was 1.3 per hour. Of the four completed 
interviews, we were able to obtain two successful Callback interviews. 

4.5.3 LI: Reference Consultation 

For this service, we discovered that of the 7 names and telephone numbers that had 
been provided to us only 3 Ind used the service recently enough to be useful. From them we 
were able to obtain one successful phone interview out of a projected four. 

4.5.4 LI: MATERIALS DELIVERY SERVICE 

For this service we only received 4 online volunteers. The remaining 22 were on-site 
volunteers. From the 26, we obtained 18 completed interviews. 

4.5.5 L2: ART AND ARCHEOLOGY - THE COLLECTION 

We opened up this study to include undergraduates from the onset and were able to 
obtain 2.19 completed interviews per hour. From the 17 Auditions we were able to obtain 15 
successful Callbacks. 

4.5.6 L2: BIOLOGY - ELECTRONIC REFERENCE 

We opened up this study to include undergraduates from the onset also and we 
obtained .92 completed interviews per hour. Of the 17 completes we were able to obtain 8 
successful Callbacks. 

4.5.7 L2: PSYCHOLOGY - REFERENCE 

We opened up this study to include undergraduates from the onset and we obtained 
1.20 completed interviews per hour. From the 16 completed Auditions we were able to 
obtain 1 1 successful Callback interviews. 

4.5.8 L3: ENHANCED ONLINE CATALOG 

For the on-site users of this service we were able to obtain 1 .30 completed interviews 
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'per hour. Of the 13 Auditions, 9 resulted in successful Callback interviews. For the online 
users who were contacted by telephone, we received a total of 45 online offers to be 
interviewed; 20 of them were from graduate students or faculty. (The remainder were 
undergraduates or alumni). We obtained 12 of the projected 13 Phone interviews. 

4.5.9 L3: SCIENCE LIBRARIES 

Document Delivery 

For this service we only received 6 volunteers (and another 5 after the closing 
deadline); of the 6, 2 were from postcards (50 had been mailed out) and the remainder were 
from consent forms distributed on site. All of them resulted completed Phone interviews. 

4.5.10 L4: ELECTRONIC REFERENCE 

For this service we obtained 25 completed on-site interviews at a completion rate of 
1.35 per hour. Of the 25 completed Auditions, .we were able to obtain 15 successful 
Callbacks. 



4.5.11 L4: INTERLIBRARY LOAN 

We obtained only 2 returned postcards out of the 50 mailed out and 1 form. From 
them we obtained two successful Phone interviews. 

4.5.12 L5: AUTOMATED REFERENCE 

This service went very well with 1.10 completed interviews per hour. Of the 17 
completed Auditions, we were able to obtain 14 successful Callbacks. 

4.5.13 L5: REFERENCE SERVICE 

This service also went well. We obtained the projected 17 Auditions at a completion 
rate of .97 per hour. We were able to Callback 13 of the 17 completed Auditions. 

4.5.14 L5: PATENT SERVICE 

As in Wave 1 this service did not go as well as anticipated. Of the 16 projected, we 
were able to obtain only 10 Auditions even though we opened up this service to 
undergraduates and other users from the outset. From the Auditions we obtained 7 
successful Callback interviews. Again, we determined that there were simply not enough 
users during the interviewing period. 
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"4.6 THE INTERVIEWERS 
4.6.1 Recruiting Interviewers 



To hire interviewers for this wave, we sent out a letter (see Appendix F) to the most 
effective interviewers from Wave I and invited them to work again. We asked them to 
contact us if they were interested. In addition, we posted the recruitment notice on the 
bulletin boards at the Rutgers School of Communication Information and Library Studies and 
asked Journalism and Library Science Faculty to announce the position in their classes. Wc 
chose not to post the recruitment notice at the cn-campus student employment offices based 
on our experience from Wave 1. 

Of the 5 interviewers from Wave 1 who were invited back, 4 were interested in 
working again. In addition, 3 new candidates were given a revised application form (see 
Appendix F). All 3 were interviewed and offered the position. Only 2 accepted. For L5, all 
of the Wave 1 interviewers were rehired. 

4.6.2 Training 

The training session for Wave II followed an outline similar to that for Wave I with a 
few modifications. During the discussion about the Questionnaire, Instructions, and Tally 
Sheet we used overhead slides to demonstrate how to fill out the documents. To demonstrate 
how to write a synopsis (see "The Questionnaire") we played recordings of 3 interviews, then 
had a trainee read aloud the examples of synopses which were handed out (See Appendix F). 
During the session, interviewers who had worked on Wave 1 were invited to interject their 
observations and comments where helpful. During the practice interviewing, the Project 
Manager paired each of the novice interviewers with one of the experienced interviewers from 
Wave 1. Because of the additional material covered, the session lasted an extra half hour. 

For L5, the on-site Project Manager conducted the training session using the materials 
and outline used at Rutgers. 

4.6.3 Reviewing 

Once again, after each interviewer's first shift, the Project Manager listened to the 
interviewer's tapes and reviewed the written Questionnaires to check the quality of the 
interviewing. Afterward, the Project Manager discussed her observations with the interviewer. 
This continued until any difficulties were resolved. Throughout the interviewing period, the 
Project Manager randomly chose a few interviews and listened to them for quality control. 
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4.7 THE RESEARCH INSTRUMENT 



4.7.1 The Questionnaires 

In developing the survey instruments for Wave 2 we had three primary objectives: the 
first was to continue quantifying the value of the service in even more detail, the second was 
to obtain an assessment of the value of the service two weeks after the person used it, and the 
third was to obtain a reading of the value of the service in relation to other University 
services. To these ends, we first developed The Audition and the Callback Questionnaires, 
drawing from the Wave I questionnaire and modifying it where appropriate, (See Appendix 
A) At the end of the Audition, we included a question asking if we could call the 
respondent back two weeks later, to explore the long term impact of the service. If the 
respondent agreed, the interviewer was instructed to turn to the last page of the questionnaire 
and record the respondent's name, telephone number, and the best time to call. The 
interviewer was instructed to write a synopsis of the user's information problem immediately 
after the interview. For services that were studied only by telephone, the Telephone 
questionnaire (see Appendix A) contained the questions from both the Audition and the 
Callback. 

After all the questionnaires were developed, we conducted a pretest at one of the 
participating libraries. During the pretest we modified some of the wording on the new 
questions. Once satisfied, we created the versions for each service using a procedure similar 
to that used in Wave 1. 

Again, all instructions to interviewers were printed in capital letters, critical 
instructions were emphasized in bold print, and all interviews were tape recorded. A set of 
detailed Questionnaire Instructions for each questionnaire was prepared. (See Appendix H), 
along with a revised set of Telephone Interviewing Instructions (see Appendix H). 

Once the Auditions were completed, we detached the last pages (Synopsis) and 
separated them into two piles. In one pile we placed the 22 Audition respondents who would 
not be called back, either because they had refused or because they had responded that the 
service was not at all helpful (a "1" or a "2") to Q.3. In the other pile, we put all those who 
were to be called back. Each Audition Synopsis was stapled to a Call Disposition Sheet and, 
where appropriate, information from the Audition Sheet was copied onto the Call Disposition 
Sheet (see Project Management section). 
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4.8 PROJECT MANAGEMENT 



4.8.1 Scheduling Interviewers 

For Wave II we only interviewed during the week, as in Wave I. To schedule the on- 
site interviewers the Project Manager calculated the number of required interviewing hours for 
each service based upon the completes per hour for Wave I and the Wave II quotas. She then 
made one list of the interviewers' preferences and availability and another of the information 
from the Impact Study Feasibility Report and the results of her conversations with the 
librarians. With primary emphasis on covering the libraries during the most productive times, 
she developed a schedule for each service. She was able to meet all needs satisfactorily. 
Again, the Project Manager was in close contact with the librarians 
at each library advising them of the interviewers' schedules and any changes. 

To schedule the telephone interviews she again divided the day into three shifts: 9- 
12:00, 1-5:00, and 6-9:00. She then made a list of each interviewer's available hours, and 
then developed a schedule (see Appendix F). For each time slot she scheduled one 
interviewer and a backup interviewer. The interviewers were advised that they were 
personally responsible for calling the backup if unable to make their shifts and were given a 
list with their colleagues names and telephone numbers. Again, each interviewer was given 
an Interviewing Assignment/Schedule and advised io phone the APLab prior to their phone 
shift to determine if there was work. 

4.8.2 Numbering the Questionnaires 

For Wave II all Audition and Telephone interviews were prenumbered. The first digit 
identified the library. For the Callback interviews the number to the corresponding Audition 
was recorded on the corresponding Callback interview, so that the Audition and the Callback 
could be matched up in data analysis. 

4.8.3 Color Coding 

To facilitate the management of the interviews, the Audition, Callback, and Telephone 
interviews were each reproduced on different colored paper: white for Audition, green for 
Callback, and yellow for Telephone. 

4.8.4 Tally Sheets 

Again, interviewers were required to keep a Tally Sheet for every shift. The Tally 
Sheet was modified slightly to facilitate the calculation of the completes per hour on a daily 
basis. (See Appendix F). 

For Wave 2, the Project Manager used an Interview Tracking Form to track the daily 
and cumulative completes per hour for each service. The form was not as helpful as 
anticipated. 

4.8.5 Call Disposition Forms 



APLAB-B%D\CLR\FINAL.REV\PROCESS.51 



Copyright (c) 1995 



March 21, 1995 P. 50 



For Wave 2 the call dispositions for Callbacks and Phone interviews were recorded 
differently than for Wave 1 . Each number to be called had its own Call C .sposition Form on 
which the dispositions of calls to that number could be recorded. (See Appendix F). As in 
Wave I, the interviewers were required to record the outcome of every call on the sheet. 

4.8.6 Tracking Callback and Phone Interviews 

To track the progress and status of the Callback and Phone interviews the Project 
Manager developed a sheet for each library with all the interview numbers for that library by 
service (see Appendix F). On the sheet she crossed out with a red pen the interview numbers 
for those who had refused to be called back or who had given a non-helpful response. She 
also crossed out any interview numbers for interviews that were missing, and eliminated any 
interview numbers for services for which we did not have enough phone numbers. Then, as 
the Callbacks or Phone interviews were completed she marked them with a pencil by putting 
an "X" through them. Thus, she was able to identify which Auditions still needed to be 
called back at any given time. 

4.8.7 Keypunching 

Once the interviews were completed, the responses to the closed-ended questions were 
keypunched and the open-ended questions transcribed. 

Along with the edited questionnaires, we again provided the keypuncher with a list of 
the questions, preferred variable names and the valid responses for each one. We eliminated 
the coding columns on this Wave because data was provided to us in a spreadsheet format. 
Once again, we asked that the keypuncher skip over the open-ended questions, punch "-1" for 
"no response" and punch each questionnaire twice to verify the entries. 

We received a floppy disk with the data along with a hard copy. Upon receiving it we 
randomly chose 10 questionnaires and checked that each was punched without any errors. 

4.8.8 Editing 

Every questionnaire was edited by the Project Manager. 
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" 4.8.9 Transcribing 



For this Wave, we asked the transcriber to revise her cost estimate from Wave 1 
because there were fewer open-ended questions on both the Auditions as well as the 
Callbacks, which she did. We gave the transcriber the interviews in three different batches, 
the Auditions first, then the Callbacks, then the Telephone interviews. Because the Callbacks 
and Auditions had the same interview number we asked that she precede all the Callback file 
names/numbers with a "CB." Once again we randomly selected a few of the transcripts and 
checked them carefully. 

Labor Costs 



The total labor cost for administering, keypunching and transcribing the Wave II 
questionnaires are outlined below: 



Task 


Quantity 


Rate 


Cost 


Interviewers 
(Training and 
Interviewing) 


223.00 hrs. 


$10/hr. 


$2,230.00 


Keypunching 


283 


.80/interview 


$ 226.40 


Transcribing 


283 


$5.45/interv. 
(average) 


$1,542.35 


TOTAL 






$3,998.75 



The average cost per interview was: $14.13. Travel costs have been omitted. Administrative 
costs are estimated as before. 40 hours, recruitment and training. 40 hours, management of 
data entry. 120 hours supervising on site interviews and 80 hours supervising telephone 
interviews. At 30% fixed and 70% variable, the 280 hours covered 14 services. 
Cost per service = 78 hrs. + N/14 * 280 
= 80 hrs. 
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2 PREPARATION FOR A STUDY 

The purpose of this manual is to assist you in conducting a cost/benefit analysis of 
user services in your library. With the explosive growth of computer and telecommunications 
technology, libraries will need to make planning and allocation decisions concerning both new 
and old modes of access to information. We have designed this study for those situations in 
which the new and old ways of providing this service are so radically different that the only 
way to assess them is by looking at their impact on the user. It is designed to assist in the 
decision-making process rather than as a study to be undertaken for its own sake. 

2.1 Recruitment 

For some of the services you will want to study (i.e. in-person reference) you will be 
able to interview users on-site at the library. In these cases, it is a good idea to complete an 
Impact Study Feasibility Report. (See Appendix F). Be aware that sometimes librarians are 
uncomfortable with the research process and may feel threatened having an interviewer in the 
reference area. Do your best to reassure them that the purpose of the study is not to evaluate 
them, but rather, to determine the costs and benefits of the service. 

For other services that are delivered electronically, or in response to a written or 
electronic request (i.e. ILL, Materials Delivery Service, etc.) you will need to recruit 
volunteers via computer or by distributing forms or postage-paid postcards at the time the 
request is made, or at the pick-up point. (See Appendix F for examples of these recruitment 
forms) and then interview the users by telephone. 

For all services it is important to determine whether there are enough users to get the 
number of completed interviews you desire. We suggest you aim for no fewer than 50 
completes per service. 

We discovered that interviews with on-site users of reference or reserve services 
(either in-person or electronic) at main libraries of large institutions yielded completion rates 
of about 1 per hour. Interviews at small, branch libraries and special collections of larger 
systems were less successful. 

For services such as materials/document delivery or ILL, for which you need to recruit 
volunteers, you will need to motivate staff to encourage users to volunteer. To obtain 
volunteers electronically through an online recruitment message, it is important to post the 
notice in a highly visible place on the menu. We were able to complete over 2 interviews per 
hour by telephone. 
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2.2 Master schedule 



Prior to undertaking this project, we suggest you prepare a Master Schedule, as 
follows: 

Task Number of Weeks 



Recruiting and hiring interviewers 6 

Writing questionnaires 6 

Pretesting questionnaires 6 

Writing instructions /Preparing forms 6 
Duplicating questionnaires, instructions, 

forms, manual, etc. 6 

Training Interviewers (including preparation) 1 

Interviewing in-person/Phone* 2 

Interviewing - Callbacks 3 

TOTAL (activities overlap) 18 



*If you are conducting phone interviews, you will need to begin recruiting 2 weeks prior to 
when you plan to begin interviewing. Thus, from start to finish the conduct of an impact 
assessment study will span 12 weeks, prior to analysis and interpretation of the data. 
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2.3 Hiring interviewers 

One of the first steps is to recruit and hire an interviewing staff. To begin, prepare a 
recruitment announcement on your letterhead. Be sure to include: 

1. A brief description of the study and its sponsor. 

2. The qualifications you are seeking. 

3. The approximate number of hours and dates the interviewer will 
work. 

4. The location. 

5. The wages you will pay. 

6. The telephone number and address to which you would like the 
interviewers to respond. 

7. Your affirmative action policy. 

Appendix F contains an example of a recruitment announcement. 

Once the announcement has been prepared, post it in areas where you think it will be 
most effective in attracting promising candidates. We had our greatest success with 
undergraduate journalism students and communications and library science graduate students. 
Those recruited via the campus-wide employment office (particularly undergraduates) proved 
less reliable, as did students of the humanities. 

Give each applicant who contacts you an application form with a letter instructing 
them where to return it and, if appropriate, a cutoff date. In addition to the standard 
questions about education and experience, we suggest including the following questions on 
the application form: 

1 . Willingness to attend the training session. 

2. Willingness to work the required number of hours. 

3. Availability. 

4. Maximum and minimum number of hours they are willing to 
work. 

5. A request for a writing sample. 

Appendix F contains examples of a cover letter and an application form. 

The next step is to schedule appointments with those who seem promising. It is a 
good idea to schedule them at least a half hour apart so you can record comments on the 
application form while they are still fresh in your mind. 

During the interview, you may wish to ask the applicant to elaborate on the following: 

Any interviewing experience (if relevant) 
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Other work experience 
Academic status 
Availability 

Preference for phone vs. in-person interviewing 

Maximum and minimum number of hours they are willing to work 

Knowledge of libraries 

Personal interests (to get a sense of their personality) 

Availability of a home telephone or email address - (this is essential so that if 
you need to contact them about a scheduling change, interviewing problem, etc. 
you can reach them) 

In addition, also note the applicant's degree of eye contact, friendliness, professionalism, 
assertiveness and overall presentation, then record any comments on the application form. 

Once all the applicants have been interviewed and you have determined how many 
interviewers you will need (see Scheduling Interviewers) select the most promising based 
upon their qualifications and presentation and send them letters offering them the position. (It 
is a good idea to hire one substitute interviewer as a backup.) In the letters, include their 
schedule, their rate of pay and what you will expect from them. (Be sure the substitute 
knows what his or her role will be.) Also include a copy of the letter for them to sign and 
return if they accept the position. Appendix F contains an example of a job offer letter. 
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3 THE METHODOLOGY/THE QUESTIONNAIRE 



To determine the long-term impact or benefit of the services, we suggest one of two 
methodologies, depending upon the nature of the service. For services in which the user can 
be approached on-site after using this service, we recommend an initial "Audition" interview 
followed by a "Callback" two weeks later. For those services for which the user is at a 
remote location or for which there is a delay between when the service is requested and when 
it is delivered (ILL, Materials Delivery, etc.), we recommend a "Phone" interview two weeks 
after the individual has used the service. This Phone Questionnaire contains questions from 
both the Audition and the Callback. (See Appendix A for examples of these Questionnaires). 

3.1 Selection of Subjects. Customization of Instruments. 

In either case, you will first need to determine who you will include in your study — 
faculty, graduate students, undergraduate students, and/or non-university affiliated users. 
Once this is determined, the Screening Question A (Q.A) on page 1 of the Audition and the 
Phone Questionnaire is modified as appropriate. For those individuals who qualify, the 
interviewer is instructed to continue on to the next question. For those individuals who do 
not, the interviewer is instructed to terminate the interview and tally the disposition (see 
below). Also, in Q.A, replace SCHOOL with the name of your institution. 

For the remainder of the questionnaire, replace $S$ with the name of the service you 
are studying. 

Q.B has been included so that you do not interview the same individual more than 

once. 

All potential respondents must be notified that you are tape recording the interview but 
that their responses will remain confidential. Then, have the interviewer record ihe interview 
number on the tape before beginning (so as to avoid any possibility of not being able to 
match the questionnaire with its corresponding tape). For in-person interviewing, we have 
found it necessary in the majority of cases to ask the respondent to hold the recorder 
themselves to ensure the tape is audible for the transcriber. 

Note that throughout the questionnaire, all interviewer instructions are printed in 
capital letters and critical instructions are emphasized in bold print. 

3.2 The Audition Interview 

At the end of the Audition, the respondents are asked if they are willing to be called 
back. If they agree, their name, telephone number and availability are recorded on the last 
page. Once the inter v iew has ended, their status (response to the screening question) and a 
synopsis of their information "problem" are also recorded and the response to Q.3 is circled in 
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the top right corner of page 1 . 



If the respondent is willing to be called back and he or she found the service at least 
somewhat helpful (a 3,4,5,6, or 7 to Q.3), the last page is detached and placed in the pile to 
be called back. 

3.3 The Callback Interview 

This is the instrument to use when calling back the qualifying Audition respondents. 
In the introduction, the interviewer inserts the service name on the first blank line and then 
describes the project from the synopsis written during the Audition. From there the Callback 
follows the questionnaire. It is numbered with the same number as its corresponding 
Audition. 

3.4 The Phone Interview 

For the Phone interviews, you will have received the names and telephone numbers of 
volunteers who contacted you electronically or by telephone. That information should either 
be transferred or stapled to a Call Disposition Form (see below). The calls are made from 
these forms and according to the questionnaire. 

3.5 A Note about the Questionnaire 

The questions contained in this Questionnaire have been created, and ordered to obtain 
information on the benefits of library services. They have been tested in numerous libraries 
and have been designed using basic survey research question writing principles. If you 
choose to add or modify a question, please keep in mind that you should refrain from using 
library jargon, be sure to avoid vocabulary that is value-laden or has double meanings, be 
careful to word questions unambiguously and use an 8th grade sentence structure. Also, be 
sure to offer respondents both options when asking a question. For exampk; "Has what you 
got from using the service on that occasion had an impact on your project oi has it had no 
impact? " Do not be concerned if questions look very similar to one another, except for an 
important phrase that differs slightly. Generally, it is a good idea to put open-ended questions 
prior to closed-ended questions. Also, keep in mind that interviewers have different styles. 
They should be expected to exercise very little judgment while administering the 
questionnaire. Any instructions should be written clearly. Finally, the physical layout of the 
questionnaire should be designed to make it as easy as possible for the interviewer to read 
and proceed from question to question. 

Once the Questionnaire has been designed, it is a good idea to pretest it on a 
subpopulation of your sample — we suggest at least two interviews with someone in your 
office or a personal acquaintance, to catch any glaring errors, a n ^ then another 10 interviews 
with actual respondents. Use that opportunity to correct any typographical errors or make any 
changes that seem appropriate. 
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Once the Questionnaire has been pretested and finalized, you will need to reproduce it. 
If you are studying more than one service, you should first give every service a number and 
pre-circle that number on the Service line at the top right hand corner of the questionnaire. If 
you are conducting the study in more than one library, you will need to assign a code for 
each Library and pre-circle the appropriate code for each one. If you are studying more than 
one aspect of a particular service, you may need to assign Version codes as well. Also, if 
you are unable to produce them with your word-processor, be sure to draw any lines with 
arrows indicating a skip pattern, prior to reproducing the questionnaire. Finally, we suggest 
that you use different colored paper for reproducing the Audition, the Callback, and the Phone 
interviews, to easily distinguish them from one another. 

Once they are reproduced, you will need to prenumber the Audition and Phone 
interviews. Starting with 1, continue on consecutively for as many interviews as you will be 
conducting. The number on the Callback interview will correspond to the number on the 
respective Audition interview. Therefore, you will need to wait until after the Callback has 
been completed to put the interview number on the Callback. 
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4 QUESTIONNAIRE INSTRUCTIONS 

For each questionnaire you will need to prepare a set of Questionnaire Instructions. 
These instructions explain to the interviewer how to administer every question on the 
questionnaire. Appendix H contains examples of instructions for the questionnaires in 
Appendix A. 
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5 PROJECT MANAGEMENT 



5.1 Human Subjects Protection 

If your institution has a policy regarding the 
Protection of Human Subjects in Research, you will need to contact them to determine the 
policy and procedures for compliance. 

5.2 Scheduling Interviewers 

In order to schedule the interviewers you will first need to determine how many 
interviews you will be completing and in what time period you expect to complete them. 
Depending upon the sample you wish to study, you may want to limit interviewing to 
weekdays. 

To schedule the in-person interviews, prepare one chart with the interviewers' 
availability and another with information about the best time to interview (from the Impact 
Study Feasibility Report). Once all this information is assembled, taking into consideration 
all the different factors, develop a schedule. 

Scheduling the telephone interviews is somewhat easier. Divide the day into three 
shifts: 9-12:00, 1-5:00, and 6-9:00. Then, review the interviewers' availability and assign 
two interviewers to each time slot -one interviewer and one backup and tell the 
interviewers it is their obligation to call their backup person if they cannot make their shift. 
Appendix F has an example of a schedule. It is a good idea to give each interviewer an 
Interviewing Assignment/Schedule. Appendix F contains an example. If there is a possibility 
that you will not have enough work for every time slot, have the interviewers phone you prior 
to their shift to ask if they should come in. 

In principle several telephone interviews can work at the same time. However, you 
must be sure that they will be acoustically isolated so that each is not distracted. 

5.2.1 Equipment; Supplies 

To tape the interviews, we recommend a portable cassette recorder. (We were very 
satisfied with the Sony Model TCM-S64V which costs approximately $30.) For the telephone 
interviews we suggest attaching a microphone to the telephone ear-piece and to a desk-top 
cassette recorder. Most electronic stores carry microphones of varying qualities. We suggest 
you purchase the best quality you can afford because it will be reflected in the recording. 

In addition to the recorders, you will need the following supplies for each interviewer: 

4 AA batteries (2 for the recorder and 2 extra) 
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A clipboard 
A name tag 
Pencils (5-10) 

A portable pencil sharpener 

Cassette tapes (1 hour tapes - 30 minutes/side) 

5.2.2 Tally Sheets 

For the in-person interviewing, every interviewer should maintain a Tally Sheet for 
every shift worked. On the Tally Sheet, the interviewers are to mark in the appropriate box, 
(with a tick mark) the outcome of every approach they make while they are interviewing, 
using the following guidelines: 

Ineligible respondent: an individual who does not qualify for the study based upon responses 
to the questions on page 1 of the questionnaire. 

Incomplete: an interview terminated by the respondent before completion. (The interviewer 
actually wrote on the questionnaire from Q.l on and then the respondent terminated). 

Interviewer terminate: an interview terminated by the interviewer because of the language 
problem, hearing problem, etc. 

Refusal: a potential respondent who refused to be interviewed after being approached. 

Complete: an interview in which the respondent's answers have been recorded on the 
questionnaire and on the tape recorder along with the interview number. 

The interviewer should bring this Tally Sheet with the completed interviews to you 
after every shift. With this Tally Sheet, you will be able to track the number of completed 
interviews per hour; this will help you to determine whether or not your are on budget. 
Appendix F contains examples of a Tally Sheet and an Interview Tracking Form for you to 
record the necessary figures. 

5.3 Reviewing 

After the first day's interviewing it is important to listen to as many of each 
interviewer's tapes as possible so that you c%n determine if he or she is administering the 
questionnaire verbatim and is probing thoroughly enough. Also, it is advisable to review the 
written questionnaires to make sure they are being completed properly. Be sure the 
corresponding number is circled, not the response itself. For example, on a yes/no question, 
the 1 or the 2 shouid be circled, not the "yes or the "no". (This will help lower your 
keypunching costs.) If you discover any problems, address them with the interviewer 
immediately so that he or she can correct them. If you have an interviewer with a particular 
problem you may want to continue this procedure for the next couple of days. If not, we 
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recommend that you take a sample of each interviewer's work on a daily basis just to be sure 
that everything is proceeding smoothly. 

5.3.1 Editing 

In addition to reviewing each interviewer's work, you will need to edit every 
questionnaire that is returned to you. This means checking to see that the questionnaire was 
filled out completely and accurately; drawing a slash through any skipped questions to help 
increase the keypuncher's speed; and coding the area study/department question. Guidelines 
for coding the area of study/department question are provided in Section 6 of this manual. 

5.3.2 Telephone Interviewing 

For the telephone interviews, the disposition of each call needs to be recorded on a 
Call Disposition Form so that you know the outcome of every call made to every respondent. 
For the Callbacks, the Call Disposition Form is on the last page of the questionnaire with the 
respondent's phone number and synopsis. For the Phone interviews, it is the sheet where the 
respondent's name and telephone number are recorded, (see Appendix F-ll). On both forms, 
there is a set of codes at the bottom for each type of outcome. You will need to prepare a 
set of Telephone Interviewing Instructions such as the ones in Appendix H5-1 and Appendix 
H5-2. 

In our experience, 30% of all contacts resulted in a completed interview. About 70% 
of all Auditions resulted in successful Callbacks. For the Callback and Phone interviews, 17- 
19% of all calls resulted in a complete. Over half (54-57%) of the calls reached an 
answering machine. 

5.3.3 Tracking, Callback and Phone Interviews 

To track the progress and status of the Callback and Phone interviews, we recommend 
that you prepare a sheet of all the intervi^v numbers for the Auditions that qualify to be 
called back. Then, as the Callback interviews are completed, cross them off so that at any 
given time you will know how many interviews are outstanding. You can do the same for 
the Phone interviews as well. Appendix F.ll contains an example of the type of sheet to 
which we are referring. 

5.3.4 Keypunching 

Once the interviewing has been completed, you will need to have the responses to the 
closed-ended questions keypunched by a professional keypuncher. To find one, we advise 
you to obtain a referral. The keypuncher will need a copy of the questionnaire, a list with 
each question and the variable name you have assigned to each question, and the acceptable 
ranges of responses for each question. In addition, you will need to tell them what to punch 
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for a "no response". We suggest a Also, we advise that you have them punch all 
entries twice in order to verify that they have been punched correctly. Finally, have them 
give you the responses in a spreadsheet format on a clearly labeled disk . Most statistical 
analysis programs can convert data from this format, and it is very convenient for checking 
and editing. Once you receive the disk, make a duplicate copy immediately and store it in a 
safe place. Our keypunching cost was $.80 per questionnaire. 

5.3.5 Transcribing 

In addition to keypunching the closed-ended questions, you will also need to have the 
responses to the open-ended questions transcribed by a professional transcriber so that you 
can code the responses. Again, it is best to obtain a referral in order to find someone 
reputable. It is important that this person type the responses verbatim and not paraphrase or 
insert any comments of his or her own. (See the end of this manual for an example of a 
transcription.) You may wish to have them transcribe a few and listen to them to make sure 
they are acceptable. We recommend that you have the transcriber give you a hard copy of 
every transcript as well as a disk with every interview labeled with: the interview number; a 
code such as "A" for Audition, "CB" for Callback or "P" for Phone; and any other relevant 
data such as the service name, etc. The transcriber's estimate will be based upon the number 
of open-ended questions; therefore, you will need to get different estimates for the Phone 
interview, the Callback, and the Audition. Our average cost for all three was $5.45 per 
interview. 



APLAB-B%D\CLR\FINAL.REV\MANUAL.51 



March 21, 1995 P, 15 



6 TRAINING 



Before the interviewing begins, the interviewers need to learn interviewing skills and 
to become familiar with the questionnaire itself. This can be accomplished in a 2 - 2 1/2 
hour training session scheduled at. a time when every interviewer (including the substitute) 
can attend. (See responses to the appropriate question on the application form). Because the 
training session is mandatory, interviewers should be paid for this time. 

For the training session you will need an overhead projector with overhead slides of 
the Questionnaire and the Tally Sheet, a tape recorder, a tape recording of a respondent 
answering question 1 on the Audition and Phone Questionnaires and the following items to 
distribute to the interviewers: 

Interviewing Training Manuals (see Appendix H) 

Examples of the questionnaires with accompanying instructions 

Tally Sheets 

Time Sheets (see Appendix F) 

Interviewer Assignments/Schedules for each interviewer 
Equipment/supplies for each interviewer 
General Rules 

An example of a synopsis (see the end of this manual.) 

While the interviewers are arriving, distribute the materials so they can review them 
prior to the session. 

Presented below is an agenda for the training session: 

6. 1 Welcome - 5 minutes 

Thank the interviewers for attending, describe the study's purpose and the 
interviewer's role. Emphasize the two main points: 

1) Probe open-ended questions. 

2) Be sure all interviews are taped and audible. 

6.2 Questionnaire Familiarization - 10 minutes 

Have interviewers pair up and, with one person playing the role of respondent and the 
other the role of interviewer, have them read half the questionnaire to their partner, then, 
midway through, switch roles. This exercise gives the attendees first hand experience with 
the Questionnaire so the remainder of the session is more meaningful to them. 

6.3 Interviewing - 45 minutes 
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Review in detail the most important concepts from the Interviewer Training Manual 
citing examples where appropriate and emphasizing the following: 

Interviewing is one of the most important aspects of the study because without it there are no 
data, and there are certain procedures an interviewer can follow to guarantee we get high 
quality, accurate and valid information. They are: 

- Develop rapport. 

- Read the questionnaire as written. 

- Know the different types of questions. 

- Probe for the most specific response. 

- Know how to handle a "don't know/refusal". 

- Edit the questionnaire. 

During the discussion about probing we suggest demonstrating the technique by asking 
some of the interviewers questions (i.e. Why did they choose their major?, Why did they 
attend the training session?, etc.) and probing their responses. It is a good idea to follow this 
up by having one interviewer question another and then critiquing their probing skills. 

Finally, have the attendees turn to page 1 of the Training Manual and, in turn, read the 
Summary Points aloud. 

Remind interviewers to read the Training Manual on their own before their first shift. 

6.4 Break - 5 minutes 

6.5 Questionnaire Instructions/Tally Sheet - 30 minutes 

The first step in reviewing the questionnaire is to call the interviewers' attention to 
their unique interviewer number which should appear on their Interviewing 
Assignment/Schedule. Also, note that each questionnaire has been prenumbered. 

Once these topics have been covered, project the overhead slide of the questionnaire. 
After reviewing the information to be recorded int the top, right, corner, have an interviewer 
read the first question from the questionnaire, then you read the instructions from the 
Questionnaire Instructions and demonstrate on the overhead how to record the response. 
Repeat for each question. 

When illustrating how to write a synopsis, play the recording of the simulated 
interview and review the example of the synopsis with the attendees. 

6.5.1 General Rules/Time Sheet - 20 minutes 

Ask each interviewer to look over the General Rules while you review them: 
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1. 

2. 
3. 
4. 
5. 
6. 
7. 



Use only pencil when interviewing. 

Be sure the microphone is near the interviewer's mouth. 

Wear your nametag at all times. 

Dress neatly/professionally. 

Try to obtain as many completes during each shift as possible. 

Be aware of screening requirements. 

Practice interviewing with a friend, spouse, roommate, etc. 



before your first shift. 

8. Write the interview numbers on the tape. 

In addition you may want to address the following: 

9. How to operate/program the tape recorder (i.e. VOR in the "Off" 
position, sensitivity on "High"). 

10. How many interviews to record on each side of the tape. (We 
recommend one per side to eliminate any chance of the tape 
running out during the interview). 

11. What to do when arriving at the interviewing site (i.e. introduce 
themselves to the lii>rarian/staff on duty). 

12. What to do if a question arises at the interviewing site (contact 
you). 

13. What to do with the completed interviews after the shift. (We 
recommend bringing them to your office no later than the 
following day.) 

14. What to do in case of an illness or emergency. 

15. For telephone interviewing: what to do in case there is no work 
(have interviewers call the day before their scheduled shift). 

Also, go over how to fill out a Time Sheet and advise interviewers that they will be 
paid after all equipment, including unused tapes, has been returned. Tell interviewers how 
and where to reach you if necessary. 

6.5.2 Practice Interviewing - 10 minutes 

Have the interviewers pair up and conduct one taped interview each with their partner. 

6.5.3 Closing 

Thank interviewers for attending. Before they leave, have them pick up their requisite 
number of cassette tapes (their quota of completed interviews divided by 2 plus 5 extra) and 
record how many they took on a sign-up sheet. 
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7 CODING RULES FOR DEPARTMENT/AREA STUDIES 



Humanities 

History' 
Classics 
Art History 
English 

Foreign Languages 
English Literature 
Film/Cinema Studies 
Religion 
Philosophy 
Drama 

Near Eastern Studies 

Musicology 

Ethnomusicology 



Social and Behavioral Science 
Sociology 

Geography (Human Ecology) 

Political Science 

Public Affairs 

Communications 

Anthropology 

Economics 

Psychology 

International Affairs 

Public Policy 

Political Economy 



Natural or Physical Science 

Biology 

PreMed 

Math 

Chemistry 

Physics 

Radiation Science 



Information Science 
Environmental Studies 
Natural Resource Ecology 

Engineering 

Any type of Engineering 

Professions 

Nursing 
Education 
Library Science 
Social Work 
Management 
Policy Management 
Business Finance 
Law 

Architecture 
Health Science 
Physical Therapy 
Leisure Studies 

Interactive Telecommunications 

Journalism 

Pharmacy 

MBA 

Museum Studies 

Criminal Justice 

School Counseling 

International Business 

Labor and Industrial Relations 

Urban Studies 

Public Administration 

Marketing 
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Other 

Undecided/Undeclared 

Music Performance 

Computer Science 

Health Economics 

Technical Communications Writing 

Nutrition 

Hispanic Civilization 
Writing 

Middle East Studies 

Policy Studies 

Speech Communication 

Public Health 

Environmental Policy 

Drug Prevention and Evaluation 

Nursing Theory Research and Development 
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APPENDIX A 
Instruments and SPSS Codes 



A.l. Wave 1 First Version 

A.2. Wave 1 . . Revised Version (1.2) 

A. 3. Wave 2 Audition 

A.4. Wave 2 Callback 

A. 5. Wave 2 Phone 

A.6. Map of SPSS variables to questions 



APLAB-B%D\CLR\FINAL.rlEV\APPSA-B.51 Copyright (c) 1995 



March 21, 1995 P. 1 



Rutgers Alexandria Project Lab Cost Benefit Study 1994 
Paul B. Kantor and Tefko Saracevic, Principal Investigators 

Appendix A 



This page intentionally blank 



APLAB-B%D\CLR\FINAL.REV\APPSA-B.51 Copyright (c) 1995 



March 21, 1995 P. 2 



I :RIC 



163 
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Appendix A 

Rutgers Alexandria Project Cost /Benefit Study 
QUESTIONNAIRE Wave I 

$A $B 

[Replace $A by name of school ] Interviewer # [1,2] 

[Replace $B by name of service] 

Interview # [3-5] 

Time Started 

Edited , 

Coded , 



Library 1 2 3 4 5 [6] 
Service 12 3 [7] 
Version 1 2 3 4 [8] 



Hello, I'm from the School of Communication, Information 

and Library Studies at Rutgers University and we are .conducting a study of 
library usage. 

A. Which of the following best describes you? (READ LIST) 

A(n) $A faculty oi staff member (GO TO Q.B) 1 

A(n) $A graduate student (SEE BELOW) 2 

. A(n) $A undergraduate student (TERMINATE AND TALLY) . . .3 
Or another type of library user (TERMINATE AND TALLY) [9] 

(LOOK TO SEE IF THERE IS A FACULTY MEMBER IN THE VICINITY. 
IF YES, TERMINATE, RECORD AND WAIT TO APPROACH FACULTY 
MEMBER . 

IF NO, AND 5 MINUTES HAVE PASSED SINCE YOUR LAST COMPLETED 
INTERVIEW, GO TO Q.B 

IF NO AND LESS THAN 5 MINUTES HAVE PASSED SINCE YOUR LAST 
COMPLETED INTERVIEW, TERMINATE AND TALLY) 

B. Have you been interviewed about using jj>B in this library within the 

past three weeks? 

Yes (TERMINATE AND TALLY) 
No (CONTINUE) 

I will need to tape record this interview. However, your identity will 
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remain completely annonymous . 

TURN ON RECORDER AND RECORD: This is interview # . 

(IF NECESSARY : ) Please would you hold this recorder? Thank You. 

3. Using a scale from 1 to 7 where 1 means not at all familiar and 

7 means very familiar, how would you rate your familiarity with this 
library? 

not at all very no 

familiar familiar response 

1 2.3 4 5 6 7 0 [10] 

4. Using a scale from 1 to 7 where 1 means no experience and 7 
means expert, how would you rate your experience with $B ? 

no no 
experience expert response 

123 4 5670 [11] 

5. How often do you use j>B during a regular semester --less than once a 

month, once a month, 2-3 times a month, 4 times a month, or more than 
4 times a month? 

Less than once a month 1 

Once a month 2 

2-3 times a month 3 

4 times a month 4 

More than 4 times a month. . .5 

No response 0 [12] 

6. Why did you use £B today? (PROBE:) Can you tell me a little more 

about that? [13-w] 

CHECK TO SEE THA^ TAPE RECORDER IS ON. 

7. Using a scale from 1 Lo 7 where 1 means not at all defined and 
7 means very well defined, in your mind, how clearly defined 
was your reason for using $B ? 

not at all very well no 

defined defined response 

1 2 3 4 5 6 7 0 [19] 

8. What did you get out of using £B today? ( PROBE : ) What would you say 
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was the value or benefit or impact of that for you? 
CHECK TO SEE THAT TAPE RECORDER IS ON. 



[20-25] 



Using a scale from 1 to 7 where 1 means not at all successful and 7 
means very successful, how successful were you in getting what you 
needed from $B ? 



not at all 
successful 
1 2 



very no 
successful response 
6 7 0 



[26] 



What would you say is the reason you feel that way? (PROBE:) 

Why else? CHECK TO SEE THAT TAPE RECORDER IS ON. [27-32] 

Using a scale from 1 to 7 where 1 means not at all confident and 7 
means extremely confident, how confident are you that you can rely on 
the information you got from using $B ? 



not at all 
confident 
1 2 



4 



extremely no 
confident response 
6 7 0 



[33] 



Using a scale from 1 to 7 where 1 means not at all difficult and 7 
means extremely difficult, please tell me how difficult was it for 
you to use $B ? 



not at all 
difficult 
1 2 



3 4 5 

About how much time did you spend using $B ? 



_minutes /hours 
1 2 



extremely no 
difficult response 
6 7 0 



no response 
0 



[34] 



[35-37,38] 



Using a scale from 1 to 7 where 1 means definitely not worth the time 
and 7 means definitely worth the time, how does the benefit you got 
from using _£B compare with the time you spent using it? 



definitely 
not worth it 
1 2 



definitely no 
worth it response 
6 7 0 



[39] 



If you were to put .a dollar value on the benefit you got from using 
$B today, what would it be? 
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$ .00 no response 

0 [40-44] 

16. If were not available at this library, is there something else you 
could have done to get this same benefit? 

Yes. (CONTINUE TO Q.17) 1 

No. (SKIP TO Q.21) 2 

No response. (SKIP TO Q.21).. 3 [45] 

17. What would you have done? (PROBE FOR SPECIFICS) CHECK TO SEE 
RECORDER IS ON. [46-51] 

18. Using a scale from 1 to 7 where 1 means not at all convenient 
and 7 means very convenient, how convenient would it have been 
for you to go somewhere else for this service? 

not at all very no 

convenient convenient response 

1 2 3 4 5 6 7 0 [52] 

19. How much time would it have taken you to get that same benefit if J>B 
were not available? 

minutes /hours /days /weeks no response 

1 2 3 4 0 [53-55,56] 

20. What would it have cost you, in dollars, to get the same benefit 
without using $B ? 

$ .00 no response 

0 [57-61] 

21. If 100 represents the best possible experience with J>B that you could 

imagine, and 0 represents the worst, what score would you give for 
the way it actually turned out? 

(THIS NUMBER MUST BE [62-64] 

BETWEEN 0 AND 100) 



22. Why did you choose that score? (PROBE:) Why else? 
I have just a few more questions. 

23. (FOR STUDENTS: ) What is your area of study? 

INTERVIEWER DISREGARD: 
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Humanities 1 

Social Science 2 

Natural Science 3 

Engineering 4 

Professions 5 

Other 6 

(FOR FACULTY / STAFF : ) With what department are you affiliated? 

INTERVIEWER DISREGARD: 

Humanities 1 

Social Science 2 

Natural Science 3 

Engineering 4 

Professions 5 

Other 6 

24. Which of the following categories best describes your age? 
(READ LIST) 

Under 18 1 

18-25 2 

26-29 3 

Thirties 4 

Forties 5 

Fifties 6 

Sixties or above 7 

(DO NOT READ) No response 0 

Thank you very much for your help . 

TURN OFF TAPE RECORDER 

******************************* 
Interviewer please record: 

Respondent ' s gender : Male 1 

Female 2 

Time ended 

Time began 

Interview length 
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Rutgers ALexandria Project Cost /Benefit Study 
QUESTIONNAIRE WAVE 1.2 



Date /94 

Interviewer # [1-2] 

Interview # [3-5] 

Time Started 



Coded 



Library 1 2 3 4 5 [6] 
Service 12 3 [7] 
Version 1 2 3 4 [8] 

Hello, I'm from the School of Communication, Information 

and Library Studies at Rutgers University and we are conducting a study of 
library usage. 

A. '.tfhich of the following best describes you? (READ LIST) 

A faculty or staff member (GO TO Q. B) 1 

A graduate student (SEE BELOW) 2 

A undergraduate student (TERMINATE & TALLY) ... .3 

Or Another type of library user (TERMINATE & TALLY) • 4 [9] 

(LOOK TO SEE IF THERE IS A FACULTY MEMBER IN THE VICINITY. 
IF YES, TERMINATE, RECORD AND WAIT TO APPROACH FACULTY 
MEMBER . 

IF NO, AND' 5 MINUTES HAVE PASSED SINCE YOUR LAST COMPLETED 
INTERVIEW, GO TO Q.B 

IF NO AND LESS THAN 5 MINUTES HAVE PASSED SINCE YOUR LAST 
COMPLETED INTERVIEW, TERMINATE AND TALLY) 

B. Have you been interviewed about using 

in this library within the past three weeks? 

Yes (TERMINATE AND TALLY) 
No (CONTINUE) 

I will need to tape record this interview. However, your identity will 
remain completely anonymous. 
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TURN ON RECORDER AND RECORD: This is interview # . 

(IF NECESSARY: ) Please would you hold this recorder? Thank You. 



Why did you use today? 

(PROBE: ) Can you tell me a little more about that? 

[10-15] 

CHECK TO SEE THAT TAPE RECORDER IS ON. 

Using a scale from 1 to 7 where 1 means not at all defined and 7 means 
very well defined, in your mind, how clearly defined was your reason 
for using today? 

not at all very well no 

defined defined response 

1 2 3 4 5 6 7 x [16] 

What is the actual project or work that brought you to use 

this service today? Please use any technical terms you may 

need, and be as specific as possible. [17-22] 

What did you get out of using 



today? (PROBE:) What would you say was the value or benefit or 
impact of that for you and for your work? 

On a scale of 1 to 7 with 1 being not at all helpful and 7 
being very helpful, how helpful was what you got to your 
actual project or work? 



[23-28] 



not at all very no 

helpful helpful response 

1 2 3 4 5 6 7 x [29] 

GO TO 0.5c GO TO 0. 6 

5a. In what way was this helpful to your project or work? [30-35] 

5b. If you had not gotten what you did, how would it have 

hurt your project or work? [36-41] 

GO TO Q.6 
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5c. In what way was it not helpful to your project or work? 

[42-47] 

5d. Did it hurt your project or work in any way? 

Yes 1 

No (SKIP TO Q. 6) 2 

No response (SKIP to Q. 6)...x [48] 

5e. How did it hurt your project or work? [49-55] 



6. Using a scale from 1 to 7 where 1 means not at all confident 
and 7 means extremely confident, how confident are you that 
you can rely on the information you got from using 
today? 

not at all extremely no 

confident confident response 

1 2 3 4 5 6 7 x [56] 

7. What would you say is the reason you feel that way? (PROBE:) 

Why else? [57-62] 

8. Using a scale from 1 to 7 where 1 means not at all difficult 
and 7 means extremely difficult, please tell me how difficult 
was it for you to use today? 

not at all extremely no 

difficult difficult response 

1 2 3 4 5 6 7 x [63] 

9. About how much time did you spend using 

today? 

minutes /hours no response 

12 X [64-66,67] 

10. Using a scale from 1 to 7 where 1 means definitely not worth 
the time and 7 means definitely worth the time, how does the 

benefit you got from using 

compare with the time you spent using it today? 

definitely definitely no 

not worth it worth it response 

1 2 3 4 5 6 7 x [68] 
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11. If you were to put a dollar value on the benefit you got from 

using today, what would it 

be? 

$ .00 no response 

x [69-73] 

12 . If were not available at 

this library, is there something else you could have done to 
get this same benefit? 

Yes. (CONTINUE TO Q.12a) 1 

No. (SKIP TO Q. 16) 2 

No response. (SKIP TO Q.16) . . .x [74] 

12a. What would you have done? (PROBE FOR SPECIFICS) [75-80] 

13 . Using a scale from 1 to 7 where 1 means not at all 
convenient and 7 means very convenient, how convenient would 

it have been for you to get this same benefit in some other way? 

not at all very no 

convenient convenient response 

123 4 567x 

[81] 

14. How much time would it have taken you to get that same benefit 

in some other way if were not 

available? 



minutes /hours /days /weeks no response 

1 2 3 4 x 



15. What would it have cost you, in dollars, to get the same 
benefit without . using ? 



[82-84,85] 



$ .00 no response 

x [86-90] 

16. If 100 represents the best possible experience with 

that you could imagine, and 0 

represents the worst, what score would you give for the way it 
actually turned out today? 

(THIS NUMBER MUST BE ^97-93; 

BETWEEN 0 AND 100) 

17. Why did you choose that score? (PROBE:) Why else? [94-98] 
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I have just a few more questions. 

18. Using a scale from 1 to 7 where 1 means not at all familiar 
and 7 means very familiar, how would you rate your familiarity 
with this library ? 

not at all very no 
familiar familiar response 

1 2 3 4 5 6 7 x [99] 

19. Using a scale from 1 to 7 where 1 means no experience and 7 
means expert, how would you rate your level of experience 
with ? 

no no 
experience expert response 

123 4 567x [100] 

20. How often do you use during 

a regular semester? (READ LIST) 

Less than once a month 1 

Once a month 2 

2-3 times a month 3 

4 times a month 4 

More than 4 times a month. . .5 

(DO NOT READ) No response x [W1] 

21. (FOR STUDENTS :) What is your area of study? 

(FOR FACULTY /STAFF: ) With what department are you affiliated? 



INTERVIEWER DISREGARD: 



Humanities 1 

Social Science 2 

Natural Science 3 

Engineering 4 

Professions 5 

Other 6 



[102] 
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22. Which of the following categories best describes your age? (READ 
LIST) 

Under 18 1 

18-25 2 

26-29 3 

Thirties 4 

Forties 5 

Fifties 6 

Sixties or above 7 [103] 

(DO NOT READ) No response x 

Thank you very much for your help. 

TURN OFF TAPE RECORDER 

******************************* 
Interviewer please record: 



Respondent ' s gender : 



Male 1 

Female 2 



[104] 



Time ended 
Time began 
Interview length. 
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Rutgers Alexandria Project Cost/Benefit Study 
QUESTIONNAIRE WAVE I I -AUDITION 



[Replace $$$ with name of service] 

Date /94 

Interviewer # 



Interview # 



Time Started. 



Library 12 3 4 5 
Service 12 3 
Version 12 3 4 

Q.3 1 2 3 4 5 6 7 -1 

Hello, I'm from the School of Communication, 

Information and Library Studies at Rutgers University and we are 
conducting a study of library usage. 

A. Which of the following best describes you? (READ LIST) 



A SCHOOL faculty or staff member (GO TO Q. B) 1 

A SCHOOL graduate student (SEE BELOW) 2 

A SCHOOL undergraduate student (TERMINATE & TALLY) 3 

Or Another type of library user (TERMINATE AND TALLY) 4 



(LOOK TO SEE IF THERE IS A FACULTY MEMBER IN THE VICINITY. 
IF YES, TERMINATE, RECORD AND WAIT TO APPROACH FACULTY 
MEMBER. 

IF NO, AND 5 MINUTES HAVE PASSED SINCE YOUR LAST COMPLETED 
INTERVIEW, GO TO Q.B 

IF NO AND LESS THAN 5 MINUTES HAVE PASSED SINCE YOUR J.AST 
COMPLETED INTERVIEW, TERMINATE AND TALLY) 

B. Have you been interviewed about using $S$ in this library within the 
past three weeks? 

Yes (TERMINATE AND TALLY) 
No (CONTINUE) 

I will need to tape record this interview. However, your responses will 



APLAB-B%D\CLR\FINAL.REV\APPSA-B.51 Copyright (c) 1995 



March 21, 1995 P. 15 



Rutgers Alexandria Project Lab Cost Benefit Study 1994 
Paul B. Kantor and Tefko Saracevic, Principal Investigators 

Appendix A 

remain completely confidential. 

TURN ON RECORDER AND RECORD: This is interview # . 



(IF NECESSARY:) Please would you hold this recorder? Thank You. 

1. What is the actual project or work that brought you to use 

$S$ today? Please use any technical terms you may need, and be as 
specific as possible. (PROBE:) Can you tell me a 
little more about that? 

CHECK TO SEE THAT RECORDER IS ON 

2. Using a scale from 1 to 7 where 1 means not at all defined and 7 

means very well defined, in your mind, how clearly defined was 
your -reason for using $S$ today? 

not at all very well no 

defined defined response 

123 4 567 -1 

3. On a scale of 1 to 7 with 1 being not at all helpful and 7 
being very helpful, how helpful was what you got to your 
actual project or work? 

not at all very no 

helpful helpful response 

123 4 567 -1 
SKIP TO 0.4 

3a. In what way was it not helpful to your project or work? 
(PROBE FOR SPECIFICS) 

3b. Did it hurt your project or work in any way? 

Yes 1 

No (SKIP TO Q. 4) 2 

No response (SKIP to Q. 4^...-i 

3c. How did it hurt your project or work? (PROBE FOR 
SPECIFICS) 

4. Using a scale from 1 to 7 where 1 means not at all confident 
and 7 means extremely confident, how confident are you that 
you can rely on the information you got from using $S$ today? 
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not at all extremely no 

confident confident response 

123 4 567 -1 

5. What would you say is the reason you feel that way? (PROBE:) 
Why else? 



Using a scale from 1 to 7 where 1 means not at all difficult 
and 7 means extremely difficult, please tell me how difficult 
was it for you to use $S$ today? 



not at all 
difficult 
1 2 



extremely no 
difficult response 
6 7-1 



3 4 5 

About how much time did you spend using $S$ today? 



_minutes/ hours 
1 2 



no response 
-1 



In a few weeks, we would like to call you with some follow-up questions 
to see how this information worked out. May I have your name and 
telephone number and the best time to reach you? 

(RECORD ON NEXT PAGE) 

*************************************************** 

Interviewer please record: 



Respondent ' s gender : 



Male 1 

Female 2 

Time ended 

Time began 



Interview length. 
RECORD THE ANSWER TO Q.3 ON T*AGE 1 
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RECORD STATUS AND WRITE A SYNOPSIS OF THE RESPONDENT'S PROJECT OR WORK 
ON THE NEXT PAGE 



Interview #. 

PLEASE WRITE CLEARLY 

Name 

Telephone number 

Best time to reach (DAYS and TIMES) : 



Thank you very much. 

TURN OFF THE TAPE RECORDER AND GO BACK TO THE QUESTIONNAIRE 
***************************************************** 

STATUS: Faculty/Staff 1 Undergraduate student 3 

Graduate Student.... 2 Another type of library user 4 

Write the synopsis here: (PLEASE WRITE CLEARLY) 
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Rutgers Alexandria Project Cost/Benefit Study 
QUESTIONNAIRE WAVE I I -CALLBACK 

m 

Date /94 

Interviewer # 



Interview # 



Time Started 



Library 12 3 4 5 
Service 12 3 
Version 12 3 4 

Hello, I'm from the School of Communication, 

Information and Library Studies at Rutgers University. A few weeks 
ago, wfl interviewed you about using $S$ to work on (DESCRIBE PROJECT) . 
We would like to ask you a few more questions about that specific 
occasion. 

I will need to tape record this interview. However, your responses will 
remain completely confidential. 

la. What did you get out of using $S$ on that occasion? (PROBE FOR 
SPECIFICS) 

CHECK TO SEE THAT RECORDER IS TURNED ON 

lb. What would you say was the value or benefit or impact of that for 
you and for your work? (PROBE:) Can you tell me a litcle more 
about that? 



On a scale of 1 to 7 with 1 being not at all helpful and 7 
being very helpful, how helpful was what you got to your 
actual project or work? 



not at all very no 

helpful helpful response 

123 4 567 -1 

GO TO 0.3 GO TO 0. 3 
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2a. In what way was this helpful to your project or work? 
(PROBE FOR SPECIFICS) 

2b. If you had not gotten what you did, how would it have 
hurt your project or work? (PROBE FOR SPECIFICS) 



3. Using a scale from 1 to 7 where 1 means definitely not worth 

the time and 7 means definitely worth the time, how does the 

benefit you got. from using $S$ compare with the time you spent 
using it on that occasion? 

definitely definitely no 

not worth it worth it response 

1 2 3 4 5 6 7 -1 

4. Has what you got from using $S$ on that occasion had an impact on 
your project or has it had no impact? 

Has had an impact (CONTINUE) 1 

Has not had an impact (SKIP TO Q.6) .2 
No response. . (SKIP TO Q.6) -1 

5. On a scale of 1 to 7 where 1 means not at all important and 7 means 
very important, how important was that impact to your project 

or work? 

not at all . very no 

important important response 

123 4 567 -1 

6. If you were to put a dollar value on the benefit of what you got 
from using $S$ , what would it be? 

$ .00 no response 

-1 

7. Have you looked into the cost or price of another way to get what you 
got from $S$ on that occasion? 

Yes 1 

No 2 

No response -1 

8. More generally, have you purchased any information services (either 

print or electronic) in connection with this project of yours? 
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Yes 1 

No 2 

No response -1 

9. For each of the following, please tell me whether you paid for the 
service with that type of funds (READ LIST AND RECORD BELOW) : 

Yes No 

Your own personal funds 1 2 

Research funds 1 2 

Funds for the support of teaching 1 2 

Other university funds 1 2 

Some other funds (PLEASE SPECIFY) ..1 2 



ASK FACULTY/ STAFF ONLY (STUDENTS SKIP TO Q.12): 

Thinking about the university in general... 



10. Have you dealt with any of the following university-wide services 
within the past year? (READ LIST AND RECORD BELOW) 

11. Using a scale from 1 to 7 where 1 means not at ail important and 

7 means very important, how important is (READ FIRST SERVICE MARKED 
"YES") to carrying out your work? (RECORD BELOW) How about (READ 
SECOND SERVICE MARKED "YES" AND RECORD RESPONSE BELOW)? (CONTINUE 
FOR EVERY SERVICE MARKED "YES") 



0. 10 



No 

Accounting 1 

Purchasing 1 

Personnel 1 

Libraries (DON'T READ FOR Q.10).l 

Maintenance 1 

Computer or Networking Center . . 1 



0. 11 



Not at 
All 

Yes Import. 

2 1 2 3 4 5 

2 1 2 3 4 5 

2 1 2 3 4 5 

2 1 2 3 4 5 

2 1 2 3 4 5 

2 1 2 3 4 5 



Very 
Import . 
6 7 



6 
6 
6 
6 
6 



7 
7 
7 
7 
7 
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Security 1 

Some other service (Please 
Specify) . 1 



I have just a few more questions. 

12. (FOR STUDENTS :) What is your area of study? (RECORD ON LINE BELOW) 

(FOR FACULTY /STAFF: ) With what department are you affiliated? 
(RECORD ON LINE BELOW) 



INTERVIEWER DISREGARD: 



Humanities 1 

Social Science 2 

Natural Science 3 

Engineering 4 

Professions 5 

Other 6 



13. Which of the following categories best describes your age? (READ 
LIST) 

Under 18 1 

18-25 2 

26-29 3 

Thirties 4 

Forties 5 

Fifties 6 

Sixties or above 7 

(DO NOT READ) No response x 



Thank you very much for your help. 
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TURN OFF TAPE RECORDER 

******************************** **********************.:*********** 

Interviewer please record: 

Time ended 

Time began 



Interview length. 
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Rutgers Alexandria Project Cost/Benefit Study 
QUESTIONNAIRE WAVE II-PHONE 



[Replace $$$'by name of service] 



Date 



Interviewer # 
Interview # 
Time Started 



794 



Library 12 3 4 5 
Service 12 3 
Version 12 3 4 



Hello, I'm 



from the School of Communication, 



Information and Library Studies at Rutgers University. We are working 
with the Director of the SCHOOL libraries to study $S$ . We hope you can 
spare a few minutes to answer some questions. 

A. Which of the following best describes you? (READ LIST) 

A SCHOOL faculty or staff member (GO TO Q. B) 1 

A SCHOOL graduate student (SEE BELOW) .......... 2 

A SCHOOL undergraduate student (TERMINATE & TALLY) ......... 3 

Or Another type of library user (TERMINATE AND TALLY) ........ 4 

B. Have you been interviewed about using $S$ within the past three 

weeks? 

Yes (TERMINATE AND TALLY) 
No (CONTINUE) 

I will need to tape record this interview. However, your responses will 
remain completely confidential. 

TURN ON RECORDER AND RECORD: This is interview # 



1. Thinking about the last time you used £S£, what was the actual 
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project or work that brought you. to use it? Please use any- 
technical terms you may need, and be as specific as possible. 
(PROBE:) Can you tell me a little more about that? 

CHECK TO SEE THAT RECORDER IS ON 

2. Using a scale from 1 to 7 where 1 means not at all defined and 7 
means very well defined, in your mind, how clearly defined was 
your reason for using $S$ on that occasion? 

not at all very well no 

defined defined response 

123 4 567 -1 

3a. What did you get out of using $S$ on that occasion? (PROBE FOR 
SPECIFICS) 

3b. What would you say was the value or benefit or impact of that for 
you and for your work? (PROBE:) Can you tell me a little more 
about that? 

4. On a scale of 1 to 7 with 1 being not at all helpful and 7 
being very helpful, how helpful was what you got to your 
actual project or work? 



not at all verv 



very no 



response 



helpful helpful 

1 2 3 4 5 6 7 -1 

GO TO 0.4c GO TO 0.4c 



4a. In what way was this helpful to your project or work? 
(PROBE FOR SPECIFICS) 

4b. If you had not gotten what you did, how would it have 
hurt your project or work? (PROBE FOR SPECIFICS) 
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4c. In what way was it not helpful to your project or work? 
(PROBE FOR SPECIFICS) 

4d. Did it hurt your project or work in any way? 

Yes 1 

No (SKIP TO Q. 5) 2 

No response (SKIP to Q. 5 )...-! 



4e. How did it hurt your project or work? (PROBE FOR 
SPECIFICS) 



Using a scale from 1 to 7 where 1 means not at all confident 
and 7 means extremely confident, how confident are you that 
you can rely on the information you got from using $S$ on that 
occasion? 



not at all 
confident 
1 2 



extremely no 
confident response 
6 7-1 



5a. What would you say is the reason you feel that way? (PROBE:) 
Why else? 

i. Using a scale from 1 to 7 where 1 means not at all difficult 
and 7 means extremely difficult, please tell me how difficult 
was it for you to use $S$ on that occasion? 



not at all 
difficult 
1 2 



3 4 5 

About how much time did you spend using $S$ ? 



.minutes /hours 
1 2 



extremely no 
difficult response 
6 7-1 



no response 
-1 



8. Using a scale from 1 to 7 where 1 means definitely not worth 
the time and 7 means definitely worth the time, how does the 
benefit you got from using $S$ compare with the time you spent 
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using it on that occasion? 

definitely definitely no 

not worth it worth it response 

1 2 3 4 .5 67 -1 

9. Has what you got from using $S$ on that occasion had an impact 

on your project or has it had no impact? 

Has had an impact (CONTINUE) 1 

Has had no impact (SKIP TO Q.ll) 2 

No response -1 

10 . On a scale of 1 to 7 where 1 means not at all important and 7 means 
very important, how important was that impact to your project 

or work? 

not at all very no 

important important response 

1 2 3 4 5 67 -1 



11. If you were to put a dollar value on the benefit you got 
from using $S$ , what would it be? 

$ .00 no response 

-1 

12. Have you looked into the cost or price of another way to get what 

you got from $S$ on that occasion? 

Yes 1 

No 2 

No response -1 

13 . More generally, have you purchased any information sources (either 
print or electronic) in connection with this project of yours? 

Yes 1 

No. (STUDENTS SKIP TO Q.17) 

(FACULTY SKIP TO Q.15) 2 

No response (STUDENTS SKIP TO Q.17) 

(FACULTY SKIP TO Q.15).-l 
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14. For each of the following, please tell me whether you paid for the 
sources with that type of funds (READ LIST AND RECORD BELOW) : 

Yes No 

Your own personal funds 1 2 

Research funds 1 2 

Funds for the support of teaching 1 2 

Other university funds 1 2 

Some other funds (PLEASE SPECIFY) ..1 2 



ASK FACULTY / STAFF ONLY (STUDENTS SKIP TO Q.17): 



Thinking about the university in general... 

15. Have you dealt with any of tae following university-wide services 
within the past year? (READ LIST AND RECORD BELOW) 

16. Using a scale from 1 to 7 where 1 means not at all important and 

7 means very important, how important is (READ W IRST SERVICE MARKED 
"YES") to carrying out your work? (RECORD BELOW J How about (READ 
SECOND SERVICE MARKED "YES " AND RECORD RESPONSE BELOW)? (CONTINUE 
FOR EVERY SERVICE MARKED "YES") 



Q. 15 



No 

Accounting 1 

Purchasing 1 

Personnel 1 

Libraries (DON'T READ FOR Q.15).l 



Yes 
2 . 
2 . 
2 . 
2 . 



Not at 

All 

Import 



1 
1 
1 
1 



2 
2 
2 
2 



Q. 16 



3 
3 
3 
3 



4 
4 
4 
4 



5 
5 
5 
5 



Very 
Import . 

7 

7 

7 

7 



6 
6 
6 
6 
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Maintenance 1 2 1 2 3 4 

Computer or Networking Center.. 1 2 1 2 3 4 

Security 1 2 1 2 3 4 

Some other service (Please 

Specify) 1 2 1 2 3 4 



6 7 
6 7 
6 7 



I have just a few more questions. 



17. (FOR STUDENTS :) What is your area of study? (RECORD ON LINE BELOW) 

(FOR FACULTY /STAFF: ) With what department are you affiliated? 
(RECORD ON LINE BELOW) 



INTERVIEWER DISREGARD: 



Humanities 1 

Social Science 2 

Natural Science 3 

Engineering 4 

Professions 5 

Other 6 



18. Which of the following categories best describes your age? (READ 
LIST) 

Under 18 1 

18-25 2 

26-29 3 

Thirties 4 

Forties 5 

Fifties 6 
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Sixties or above 7 

(DO NOT READ) No response -1 

Thank you very much for your help. 

TURN OFF TAPE RECORDER 

****************************** 
Interviewer please record: 

Respondent's gender: Male 1 

Female 2 

Time ended 

Time began 

Interview length 
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A.6. Code Book For the SPSS Data Set. The following tables show the correspondence between 
specific code names assigned in the data sets and the questions on specific instruments. 
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First Wave First Version (1.1) 



Variable 


Question 




Name 





Description 


lil 1 V 






SESSN 




Tntprvipw ff 


LIBRY 




T vibrarv 

1 j 1 l/ 1 14-1 y 


SFRV 




Jul V 


VFRSN 




V CI olUli 


STATUS 


Ol 

V 1 




FAMT IR 


v- 5 


Prtmiliaritv with lihrarv 

X 0.111111C11 1L y Willi XlUlOi y 


FAMSFR 


04 


Fyt^pHpupp with rpfprpnpp QPrvipPQ fT? 

X_/A.L/^/l IWtlw^ Willi 1 ClCl tfll^/t; Vl^/t/J ^1\0 ^ 


FRFSFR 


05 

v J 


jnuw uncii uu yuu UoC ocrvicco 


V 
A 




AX/hv HiH unii iica T?^ *nHiiv 
vviiy uiu yuu UoC ixo uuuay 


CT DFF 


07 


fMaritv nf Hpfinifinn for limine 

V_^la.lliy Ul LlC.lillli.IUil 1U1 Uolilg xxo 


GETOUT 


Q8 


What did you get out of using RS today 


SUCCES 


Q9 


Success at getting what you wanted from 


X 


Q10 


Reason you feel that way 


CONINF 


Qll 


Confidence in reliance in info from RS 


DIFFIC 


Q12 


Difficulty in using RS 


TIME 


Q13a 


Time spent 


TMUNIT 


Q13b 


Unit of the time 


TIMBEN 


Q14 


Benefit of use versus time spent using RS 


VALDOL 


Q15 


Dollar value (whole $$$ only?) 


ALTWAY 


Q16 


If RS not available, someplace else? 


X 


Q17 


What would you have done? 


ALTCON 


Q18 


Convenience to go elsewhere 


ALTTIM 


Q19a 


Time elsewhere for benefit if RS unavail. 


ALTTMU 


Q19b 


Unit of the time 


Q20 


Cost to get benefit eleswhex-e (whole $$$) GRADE 


Experience with RS rating 




X 


Q22 


Why did you choose score for (21)? 


AREAST 


Q23a 


Area of study (Students Only) 


DEPTMT 


Q23b 


Area of study (Faculty/ Staff Only) 


AGEGRP 


Q24a 


Best describes your age 


GENDER 


Q24b 


Gender 



APLAB-B%D\CLR\FINALREV\APPSA-B.51 Copyright (c) 1995 



March 



Rutgers Alexandria Project Lab Cost Benefit Study 1994 
Paul B. Kantor and Tefko Saracevic, Principal Investigators 

Appendix A 

Mapping of SPSS variables to questions. 

Wave 1.2 Instrument 



Variable Name 


Question Number 


Date 


Date 


INTV 


Interviewer # 


SESSN 


Interview # 

JLlllXsl V lv W TV 


LIBRY 


LIBRARY 

J .• JL L* Ivy VAX JL 


SFRV 


SFRVirF 

01_/I\. V 1V^J_/ 


VERSN 


VFRSTON 


STATUS 




Y 
A. 


V 1 


fT RDFF 




X 


o^ 


x 


04 


HELPFL 


05 


x 


Q5a 


x 


Q5b 


x 


Q5c 


HURTFL 


Q5d 


X 


Q5e 


CONINF 


06 


X 


07 


DIFFIC 


Q8 


TIME 


Q9a 


TMUNIT 


Q9b 


TIMBEN 


Q10 


VALDOL 


Qll 


ALTWAY 


Q12 


X 


Q12a 


ALTCON 


Q13 


ALTTIM 


Q14a 


ALTTMU 


Q14b 


ALTDOL 


Q15 


GRADE 


Q16 


X 


Q17 


FAMLIB 


Q18 


FAMSER 


Q19 


FRESER 


Q20 
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AREAST 
DEPTMT 
AGEGRP 
GENDER 



Q21a 
Q21b 
Q22 

GENDER 
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Instrument: Audition 



V allaUlu l^ailic 


Onp^tinn NTnriihpr 


Plat** 


JL-/U.LC 


TMTV 
UN 1 V 


T n ton /i avi jo r* Xt 

interviewer tr 




111LC1 VACW Tr 


T TOR V 


T in t*o n / 
jUI Diaiy 


otK V 




VFRSN 


VprQinn 


X 


Q3 Recap 


Status 


Best Describes 


X 


Qi 


CLRDEF 


Q2 


HELPFL 


Q3 


X 


Q3a 


HURTFL 


Q3b 


X 


Q3c 


CONINF 


Q4 


X 


Q5 


DIFFIC 


Q6 


TIME 


Q7a 


TMUNIT 


Q7b 


GENDER 


GENDER 


LENGTH 


Interview Length 



An "x" indicates an openended response which was not directly coded by the keypuch service. Those 
codes were later assigned. Since we permitted up to 3 codes to be assigned to each open ended 
response, the variables were created in groups of three. 
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Instrument: Callback 



ValldLUL. iNalUc 


III 1A Ctl Ptf"l MllTT\r\At* 


JLyalc 




UN 1 V 


interviewer ft 


w<;<;n 


Interview # 


T TRRY 


jlj Diary 


o jC/Iv V 


ocrvit-c 


VCD CM 

V CixolN 


v ersion 


X 


via 


X 


nin 
v<j o 


ucr per 




X 


v^za 


X 


V^ZD 


1 11VJ.I5C.1N 


\P 


TMP APT 
liVlr /VL, 1 




IMPORT 


OS 


VT DOT 


Ofi 


AMWAY 

All VV A X 


07 


PIJRCHTNF 


OX 


PFRSFTIN 


OQa 


RESFUN 


09h 


TEACHFUN 


OQr 


UNIVFUN 

V_/ ill T Jt V_/ L l 


09H 


OTHERFUN 


Q9e 


ACCOUNT 


QlOa 


PURCHAS 


QlOb 


PERSON 


QlOc 


LIBRAR 


QlOd 


MAINTEN 


QlOe 


COMPUT 


QlOf 


OTHERSRV 


QlOg 


ACCTIMP 


Ql la 


PURCHIMP 


Ql lb 


PERSIMP 


Qllc 


LIBRIMP 


Ql Id 


MAIN IMP 


Qllc 


COMPIMP 


Ql If 


OTHERIMP 


QHg 
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AREAST Q12a 

DEPTMT Q12b 

AGEGRP Q13 

LENGTH Interview Length 
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Instrument: Phone 



Variable Name 



Question Number 



Date 

INTV 

SESSN 

LIBRY 

SERV 

VERSN 

Status 

x 

CLRDEF 

x 
x 

HELPFL 

x 

X 
X 

HURTFL 

x 

CONINF 
x 

DIFFIC 

TIME 

TMUNIT 

TIMBEN 

IMPACT 

IMPORT 

VALDOL 

ANWAY 

PURCHINF 

PERSFUN 

RESFUN 

TEACHFUN 

UNIVFUN 

OTHERFUN 

ACCOUNT 

PURCHAS 

PERSON 



Date 

Interviewer # 
Interview # 
Library 
Service 
Version 



Best Describes 

Ql 

Q2 

Q3a 

Q3b 

Q4 

Q4a 

Q4b 

Q4c 

Q4d 

Q4e 

Q5 

Q5a 

Q6 

Q7a 

Q7b 

Q8 

Q9 

Q10 

Qll 

Ql2 

Ql3 

Q14a 

Q14b 

Q14c 

Q14d 

Q14e 

Q15a 

Ql5b 

Q15c 
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LIBRAR 


Q15d 


MAINTEN 


Q15e 


COMPUT 




SFCTJR 




OTHFRSRV 


1 will 


ACCTIMP 

1 !■ VV J. J I III 


016a 


PURCHIMP 


Q16b 


PERSIMP 


Q16c 


LIBRIMP 


Q16d 


MAINIMP 


Q16e 


COMPIMP 


Q16f 


SECURIMP 


Q16g 


OTHERIMP 


Q16h 


AREAST 


Q17a 


DEPTMT 


Q17b 


AGEGRP 


Q18 


GENDER 


GENDER 


LENGTH 


Interview Length 
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Assignment of Session Numbers fur Interviews in Wave 2 

1-4 



Info Desk 
Ref. Consult 
Brl REF 
Br2 REF 

Mat. Delivery Serv. 



5-8 



17-34 



9-12 
13-16 



Callback 
Phone 

Callback 
Callback 



Phone 



Art and Architecture 101-117 

Biology 118-134 

Psychology 135-150 



Callback 

Callback 
Callback 



Reference 
Interlibrary Loan 



201-225 
226-230 



Callback 



Phone 



OnlineCat Phone 301-313 
OnlineCat Site 314-326 
Scien e Doc. Delivery 327-331 



Phone 



Callback 



Phone 



Patents 
Reference 
Auto Ref. 



401-416 
417-433 
434-450 



Callback 
Callback 
Callback 
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APPENDIX B 



Open Ended Coding 

Bl. The Empirical Taxonomy 

Definitions of Codes, Example Detailed Instructions 

B2. Map of Codes to Questions 

B3. Map of Empirical Codes Derived Codes 
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B 1 . Coding Scheme (Empirical Taxonomy) derived from the open ended responses. ' This 
scheme was developed primarily by Mr. Michael Wilk, working under the direction of Paul 
Kantor. Details of the process are given in Chapter 4 of the Report. 



Detailed coding scheme for open ended responses. We present here the several scales 
that were developed and used. For example, the scale called "Reason" was applied to 
questions: Wave 1.2 Question 7 (W1.2Q7), to Wave 2 Phone Question 5 (W2PQ5) and so 
forth. The text of the question in each case assessed "What was the reason you gave the 
score you did regarding the confidence in relying on the information you got from the 



service / 



In the SPSS analysis of multiple responses, each response must be assigned an integer 
label. Those labels, which appear in the discussions in Chapter 4, are shown at the right 
margin. Thus, for the class "Choose", the response L3, "librarians/people were not helpful" 
was numerically coded as "13". 

Choose: W1.2Q17. Why did you choose the score you did for the service ycu used? 
Librarians/People 

LI - librarians/people, I was having problems getting my point across 11 

L2 - librarians/people were helpful finding what I was looking for 12 

L3 - librarians/people were NOT helpful finding what I was looking for 13 

L4 - they (Librarians/people) weren't in my area of specialization 14 

Time & Money & Efficiency 

Ql - saved time, quickly 21 

Q2 - took too much time 22 

Q3 - saved money 23 

Q4 - more efficient, more easy, convenient, access from home 24 

PR - Personal Reasons 

PR1 - gave me a direction 30 

PR2 - look it up myself 31 

Physical Environment 

PI - good physical environment 41 

P2 - bad physical environment 42 

Got Something 
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Gl - got everything I wanted, needed, more than I thought, satisfied me , helpful 51 

G2 - got MOST of the information 52 

G3 - didn't get everything I needed, could have been more, couldn't give me 

everything I needed or wanted, no benefit 53 

Equipment Performance 

El - equipment does work well, easy to work with, well designed, is user friendly, 

fun 61 

E2 - equipment doesn't work well, not easy to work with, not users friendly 62 

E3 - I'm still learning how to use the equipment, the system 63 
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Done: -W1Q17. Something else you could have done to get same benefit? 



1 - Do Nothing, no, stayed with current information nothing, stuck with, without them, 

don't know 1 

2 - Oo more on my own in the Library (unassisted or manually) 

2A - gone through indexes 21 

2B - randomly look around 22 

2C - used an on-line (electronic) service 23 

2D - gone through abstracts 24 

2E - gone through readers guide 25 

2F - manually used card catalog 26 

2G - could have checked availability of book, check book out, browse the shelves 27 

3 - Use other Library Services: 

3A - used interlibrary loan or recall 31 

3B - librarian/info desk/reference librarian 32 

4 - Could have used another library 

4A - could have used a Public Library 41 

4B - could have used another University/College library 42 

4C - by checking the computer system to see if it's available 43 

5 - Could have gone to or contacted another place (OTHER THAN LIBRARY) 

5 A - bookstore/record store/video store 51 

5A1 - buy items (books, tapes, videos, magazines) 511 

5A2 - rent items (books, tapes, videos, magazines) 512 

5B - another computer facility 52 

5C - my place of work, through the company, my department 53 

6 - Could have contacted PEOPLE outside the Library about where to go, see if I 

could borrow from 

6 A - friend 61 

6B - professor/advisor/teacher 62 

7 - Other or miscellaneous 7 
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Done2: W1.2Q12A. If the service were not available at this library, is the 

re something else you could have done to get the same benefit? What would you have done? 



P - had some people help them by talking to people somewhere 10 

PI - ask the reference desk/librarian/info desk 11 

P2 - learned it in class 12 

P3 - called the company 13 

P4 - omitted 14 

P5 - asked professor/advisor 15 

P6 - pay someone money to do it 16 

M - do it myself 20 

Ml - did it by trial and error 21 

M2 - spent time on the computer doing it myself 22 

M3 - do nothing 23 

I - use another information source: 30 

II - information source: library of congress 31 

12 - information source: On-line service ■. 32 

13 - information source: newspapers, periodicals, textbook 33 

14 - information source:Victor/card catalogue 34 

15 - information source: friend 35 

16 - information source: Public or unspecified Library 36 

17 - information source: University Library or University 37 

18 - information source: abstracts 38 

19 - different form of the information source : (i.e. print version, etc.) 39 

110- a place that points to another information source (i.e. Reference books, articles, 

books, indexes in books, etc.) 40 

Rl - Use another research method 50 

T - travel to another place (not specific) to use another or same kind of source: ... 60 

Tl - omitted 61 

T2 - omitted 62 

T3 - another agency ( not library) 63 

T4 - another city/or city 64 

LI - use inter-library loan 70 
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Hurt: W1.2Q5B, W1.2Q5C, W1.2Q5E, W2CQ2B, W2PQ4B, W2PQ4C, W2PQ4E, 
W2AQ3A, W2AQ3C. If you had not gotten what you did, how would it have hurt your 
project or work? 

Nl - not been able to do it, not been able to complete project, not done project as well, not 



helpful, been returned to me, nothing to back me ■* 10 

SI - would have to talk or contact someone or figure it out myself 20 

I - information 30 

II - gotten less or older information, less useful, less effective, different, wasn't as good . 31 

12 - not gotten information/answer at all 32 

13 - Use same information/resources in a different form or another way 33 

14 - gone to another information source 34 

Ml - (money) losing money 40 

Tl - (time) Would have taken more time, delayed, get behind, procrastinate 50 

P- Personal Reason 60 

PI - not sure, don't know 61 

P2 - I'm in trouble, really hard, hurt a great deal 62 

P3 - not knowing where to start, stumbling around in the dark 63 

Gl - would have gone physically somewhere else, somewhere else (not specified) 70 

H2 - it wasn't important, didn't hurt project, Wouldn't hurt, just know I need to do mor; 

work ." 80 
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Reason: W1.2Q7, W2PQ5, W2AQ5. What was the reason you gave the score you did regarding the 
confidence in relying on the information you got from the service? 



I - INFORMATION ITSELF 10 

II - information is historical facts not opinions 11 

12 - information is specialized, related to my topic, detailed information 12 

13 - information is general 13 

14 - omitted 14 

15 - information comes from another source, that same information is used elsewhere, double 

check it 15 

16 - allows me to pursue to other places, heads me to other info, narrows down the amount I have 

to read, without it couldn't find what I need 16 

17 - NOT up-to-date, possibility of error 17 



P - Personal Reason 20 

PI - successful in finding EVERYTHING they needed 21 

P2 - fairly successful in finding MOST of what they needed 22 

P3 - WASN'T successful in finding what they needed 23 

P4 - DON'l KNOW, NOT SURE, will see if it works, need to talk to my teacher 24 

P5 - experience tells me I got most of what I needed 25 

P6 - I have confidence up to a certain date, is up-to-date 26 

Staff Performance 

Good Characteristics 

E2 - helpful, showed me how, they're competent 32 

E3 - they were comfortable with the information 33 

BAD Characteristics 

E5 - not helpful, they're not competent 35 

E6 - they were not comfortable with the information 36 



INFORMATION SERVICE 

GOOD Characteristics 

52 - the reputation/sources of the service is good and reliable - I'm confident in it, it's accurate ... 42 

53 - have used the service in the past, its been useful, accurate and has worked 43 

54 - it's easily accessible, user friendly and convenient 44 

55 - it's fast, saves time 45 

56 - a lot of information available, more was provided than I thought possible, variety of sources, 

gives choice 46 

57 - still Learning how to ucc it 47 

58 - I have faith, trust, was given to us 48 
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S9 - BAD Characteristics 

S10 - sources are NOT complete, NOT comprehensive, NOT useful, Not confident, not enough . . . 
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Way: W1.2Q5A, W2CQ2A, W2PQ4A, W2AQ3A. In what way was this helpful to 
your project or work? 



H - helped me/very helpful 10 

H 1 - helped nie to find 11 

H2 - helped me to find the information I needed (not specific) 12 

H3 - helped me to find the information I needed for a task 13 

H4 - helped me start, Gave me more direction about where to start (not specific), 

narrowed, down search 14 



G - gave me something 20 

Gl - gave me a better grade 21 

G2 - gave me ideas, insight, perspective 22 

G3 - allowed me to complete or helped my project/ wouldn't have gotten it done 

without it/had to do it 23 

G4 - gave me information/references 24 

G5 - gave me up-to-date information 25 

G6 - gave me access to people so I can get more information 26 

T - time 30 

Tl - saved time , didn't have to sit around 31 

T2 - did not save time 32 

Ml - save money 40 

N - did not find anything 50 

D - don't know anything 60 
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Getout: W1Q6, W1Q8, W1Q10, W1Q22, W2PQ1, W2PQ3A, W2PQ3B, W2AQ1, W2CQIA, W2CQ1B, 
W1.2Q1, W1.2Q3, W1.2Q4. What did you get out of using the service? What was the value or benefit or 
impact? 

1 Orient 1 

3 A to locations of physical items in Library 11 

IB availability of resources or how much material is accessible, call numbers and type of materials: broad 

overview 12 

1C learn how to use equipment or resources items in library, facility, i.e. computer systems, electronic 

resources 13 

2 Work 2 

2A it is their place for work, their carrels are there 21 

2B to use the computer 22 

3 Get 3 

3A a physical Object like a book, tape, periodical etc 31 

3B information, knowledge, background info, analysis, answer a question, clarification, made clear, find out, 

listen to something, view a film 32 

3C save time, quickly 33 

3D convenience, efficiency, make easy, , no problem, no trouble 34 

3E save money, didn't have to pay, cost effective 35 

3F fulfilled goals, task completed, got what I wanted, got everything I was looking for\even more than I 

expected 36 

4 did not get 4 

4A item that wasn't available or had to go somewhere else to get it 41 

4B did not save time, too. too long, longer than expected, time waiting, turnaround time isn't fast 42 

4C was not convenient, not easy to use, frustrating, not user friendly, Equipment didn't work well or not 

designed well 43 

4D didn't fulfill goals, didn't get everything I wanted, not able to gel/see everything, not enough of, no 

access, incomplcte.t current, there could/should more 44 

Reason Codes : 

RT For Task Accomplishment: 5 

RT1 for research 51 

RT2 for a paper, dissertation, thesis, report, project 52 

RT3 to make bibliography more complete 53 

RT4 to put (NOT GET) a book, film, video on reserve 54 

RT5 It's part of my job 55 

RT6 for a class, studies 56 

RT7 stay current, catch-up, recent 57 

RT8 to point to another source of information 58 
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RT9 allows access to inaccessible materials, go and get later, it's available 59 

RS For Someone 6 

RSI for a professor 61 

RS2 for a student(s) 62 

RP For Personal Reasons 7 

RP1 for leisure, pleasure, personal interest (not related to work), browsing 71 

RP2 freedom from worry, reduces stress, makes teaching easier 72 

RP3 didn't want to travel, want to acces from home 73 

RE Library Environment 8 

RL Library Physical Layout 81 

RL1 could be better 811 

RL2 not big enough 812 

RL3 material not organized well . . . . , 813 

RR Procedure/Rules 82 

RR1 Fines too high 821 

RR2 Check out books for longer periods of time, time limits 822 

RR3 Library doesn't accept delivers 823 

RP Performance 83 

RPE Equipment performance 

RPE1 Worked well, user friendly 931 

RPE2 Didn'l work well, not designed well, not user friendly 932 

RPS Staff Performance 824 

RPS1 Positive (people were nice, helpful, good at job, efficient did what they were supposed 

to do) 941 

RPS2 Negative (was not notified, had difficulties, unskilled clerks) 942 
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Example Coding Instructions as they were used during the development and testing of the 
Empirical Taxonomy. These are the instructions for the "Getout" class of responses. 



Questions 6, 8, 10 & 22 Coding Instructions 



A. Read the entire response first before coding. Ask yourself : Do yoii understand what 
they're saying? Do their responses make any sense within the context of the original 
question? Reread the response and question. Can you recognize any keywords in the 
response that match the keywords for a code? Long and confused answers frequently contain 
only one or two concepts. Try to identify the concept(s) arid corresponding code(s) by asking 
what action did this person say they would take and what was their reason for taking it. 
What is the main theme of their response? 

B. Concentrate on the last things they say, usually it's the most meaningful part of the 
answer and what they're really trying to say. If there's nothing to code there, pay attention 
to the first thing they say. The initial response frequently contains codable information. 



CODING RULES: 



A. You may use up to three (3) codes per response only. 

B. Code only for that specific occasion. Do not code when the respondent speculates on 
things they "might" do or their experiences of the past. It should be clear this is the action 
they would immediately carry out. If the respondent only speculates about what they would 
do, treat it as a legitimate response and code it. 

C. If you're not sure, don't code it. Code only those ideas that are clear following your 
reading of their response. 

D. Most responses will contain an action and a reason for the action. This is how the code 
sheet is divided, between action codes and reason codes. Action codes are 1 - 4, reason 
codes are those with the "R" in front of them. Learn how to recognize their patterns. The 
following words frequently indicate the presence of actions and reasons: 



Action 
get 

getting out 
I got 
I needed 
I found 
looking for 
it helped me 



Reason 

for my_ 
I'm doing for 
It helped in my_ 
I needed to 



Ask yourself what are they trying to do or get? This is the action part of the response. Then 
ask why are they committing this action? What is the reason behind their action? 
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Respondents will say they do something, get something, watch something, and follow it with 
a reason. If the response is not coded in the reason section look to the action section for a 
code that is appropriate. 

Sometimes they will answer with reasons only. In this case., be on the look out for "and". 
This will signal the presence of two or more reasons for what they're doing. Code all the 
reasons you have space for - it's the information this study is most interested in. 

When the respondents use the following words: check, see, locate and find - all of these can 
be interpreted as getting something or taking an action. If thev gain information, time, 
convenience, money or unfulfilled goals - all can be considered as getting something (action). 
The point here is to not confuse actions with reasons. There will be responses however that 
sound like a reason and can only be coded from the action codes - code them that. way. The 
only rule is that there are no hard and fast rules. We want to "capture" the information as 
accurately as possible by whatever code does the trick. These are action responses which are 
different from reason responses. Occasionally a respondent might say "getting some 
research". The problem coding this one is that the "getting" something, an action, is 
"research" which is a reason code. Code the verbatim only. Research is the only code clearly 
present - code it. Always defer to the verbatim rule (see E). 

E. As a general rule: code using the verbatim responses, that is , if you don't see it in print, 
don't code it. You should guard against "reading into" a response too much. Code only 
what's there by either matching the verbatim or picking out the one or two concepts that are 
being communicated in the response. If they use "and", then look for multiple reasons 

CODING PRIORITIES: 

A. . Try to code to the most specific level possible. If you cannot code to that level of 
specificity, use the general level or Boldface codes. 

B. Code all reasons first followed by actions. What respondents say they are doing is not as 
important as to why they are doing it. They will get "something" for some reason. Code the 
reason or reasons first then, if space remains from your three (3) limit code response, code 
the action. 

C. Always code the last thing they said. Then work your way back. As an example, a 
response such as "research project" should be first coded as a "project", "research" second. 
Clearly they're both reasons, so if only one unused code space remains you would choose 
"project" as the last code. The same is true for "class project", code for "project" first, 
"class" second. However, for sequence of actions, see D below. 

D. All actions should be coded in the order they arc given by the respondent. If they say: "1 
first would do ACTION X, then I would do ACTION Y followed by ACTION Z". Your 
coding priority should correspond to the chronological sequence of the respondent's actions. 
So your first action code would correspond to ACTION X, then Y and finally Z - assuming 
you still have not used all three of your coding slots. 



:RIC 
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Code Descriptions 

1 Orient 

1A To locations of physical items in Library 

IB Availability of resources or how much material is accessible, call numbers and type of 
materials: broad overview 

1C Learn how to use equipment or resources items in library, facility, i.e. computer systems, 
electronic resources 

Keywords 

1A: to find, find out, located at, help, where, didn't know, trying to find/look, searching, 
where to look, 

IB: familiar with services, resources available, looking for sources, oriented, start, where 
1C: Learn, taught me how to use, showed me how to, told how, familiarity with using 

Code l's: These codes are concerned with the respondent's orientation and familiarity with 
items in the library. Look for "find" words when trying to code category 1 codes. 

Code 1A: Respondents tell of their gaining familiarity with the locations of things like 
copiers, computers, book shelves, layout of the building etc. This code's main theme is the 
location of things within the library - where things are. 

Code lB's theme is respondent's discoveries of what the library has available to them in the 
way of books, periodicals and services. Not concerned with where these items are (that's 
code 1A), they are intrigued by what opportunities the library has to offer. 

Code lC's theme is with respondent gaining experience, or being taught how to use some of 
the libraries' resources, usually equipment. They learned how to use electronic catalogues, 
computer systems, copier machines, equipment that gave them access to documents. 

2 Work 

2A It is their place for work, their carrels are there 
2B To use the computer 

Keywords 

My carrel/desk is there, I come to do my work here, able to work here, used computer 

Code 2A is concerned primarily with students who come there to do their studies. Their 
carrel or desk is there so that's why they came to the library that day. It is a part of their 
normal everyday work activities. 

Code 2B's theme is about students who came to the library to use the computer for writing a 
paper or for some other project. 

3 Get 
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3A a physical Object like a book, tape, periodical etc. 

3B information , knowledge, obtain background info, analysis, answer a question, 
clarification, made clear, find out, listen to something 
3C save time, quickly 

3D convenience, efficiency, make easy, user friendly, no problem, no trouble 
3E save money, didn't have to pay, cost effective 

3F Fulfilled goals, task completed, got what I wanted, got everything I was looking forgiven 
more than I expected 

Keywords 

Get, got; check see, locate, saved, easy, convenient, fast, simple, Objects include: books, 
periodical, journal articles, information, resource , tape, material, item. 

This is the "action" part of the response and will frequently be present. Become familiar and 
learn how to recognize these codes - you will see them allot. Reread Coding Rule Paragraph 
D about action and reason codes again. 

Code 3A is identified when the respondent says they came to the library to get a physical 
object like a book, periodicals or magazine. Although seemingly clear at first glance this 
code can frequently cause trouble. Sometimes it is not altogether clear what exactly is a 
physical object. You'll see words like reference, item or material and an action associated 
with looking at these things but with little indication of whether it's information or the actual 
books/objects they're after. In general, code 3A any time objects like books, references , 
periodicals, tapes, music, film or anything physical is mentioned that they're trying to get, had 
used or are looking for. If there's an item they're looking for, there's a 3 A code present. 

Code 3B's theme is something that is intangible, you can't touch it and no mention of a 
physical object is present in their icsponse. The respondent is interested in reading, analysis, 
answering a question, subject knowledge, checking facts, listing to something - no where do 
they mention a physical item. Also information can be call #s, titles, citations, literature or 
information leading to the location of an items like references that lead them to an object. If 
it's clear they're trying to get a book through information, code both object 3A and 
information 3B. Stick to the verbatim rule - Paragraph E under Coding Rules. 

Code 3C's respondents are interested in saving time, getting it done quickly - it was fast, and 
didn't take too long. Happening right away, promptly, within their time frame, didn't waste 
time are the concerns of 3C's respondents. 

Code 3D's theme is ease of use, convenience, having little trouble with things. These 
respondents got what they wanted with little hassle and inconvenience. Their experiences will 
be described as effective and simple. Also convenience can be coded as narrowing down a 
search. Don't use this code for responses related to equipment performance, see Code R17 
for that. Be careful you don't confuse this code with the time saving code 3C, they're very 
similar. You will see 3C's and 3D's frequently together in the same response. 

Code 3E's respondents are the money saving people. They didn't have to buy something or it 
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was cost effective. In short, saving money is the one theme here and is generally easy to 
identify. 

Code 3F can be identified when the respondent says they got what they wanted and needed/ 
what they were looking for, without any mention of getting information, an object or anything 
else. Although not a useful code because it doesn't tell us much, it frequently appears. The 
respondents', in effect, don't tell us much more than "yes, I got what I wanted" nothing more, 
nothing less. They completed what they set out do and sometimes got even more then they 
asked for, beyond their expectations. There's a sense of task completion. Again, if you can 
find another code that is more specific use it but many times this is all you'll have. 

4 Did not get 

4A Item that wasn't available or had to go somewhere else to get it 

4B Did not save time, took too long, longer than expected, time waiting, turnaround time isn't 
fast 

4C Was not convenient, not easy to use, frustrating, not user friendly, long time to use, didn't 
have to travel 

4D Didn't fulfill goals, didn't get everything I wanted, not able to get/see everything, not 
enough of, no access 

Keywords 

Did not get what they wanted/needed, not worthwhile, not useful, didn't find, didn't get 
access, missing, not user friendly, hard to use, too long, didn't save time, not enough of 

Cor'c 4A will frequently b r -ecognized in the interlibrary loan questions. Respondents were 
trying to get a book or item at a library and it wasn't there necessitating them to use the 
interlibrary loan or Materials delivery service. They also could have been looking for an item 
and it just wasn't there prompting disappointment from the respondent. Be careful not to 
code these responses as convenience. If they're getting it from somewhere else think of it as 
"not being available" from where they would normally get it. 

Code 4B is the exact opposite of Code 3C. This code will appear whenever the respondent 
says it took too long, longer than they originally thought or expected. It just wasn't fast 
enough for these people. They didn't save time - they lost and wasted time 

Code 4C is the exact opposite of Code 3D. These respondents experiences with this service 
were not easy. Their experience was not convenient. Be careful not to confuse this code 
with equipment performance not being up to par - use Code R17 for that response. 

Code 4D is the exact opposite of Code 3F. Here the respondent did not get their goals 
fulfilled, were not able to get what they wanted. There should have been more for these 
people, it wasn't complete and not current. A general uncertainty pervades their response; it's 
as if they didn't see everything they hoped to see or get. They will mention how incomplete 
their experience was or what they got wasn't current. 
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Reason Codes Rl - R 17 
Keywords 

"for my", "I'm doing for", "It helped in my", "I needed to" 

Please reread Coding Rules, Paragraph D to fully understand the difference between reasons 
and actions. Essentially a reason can be identified by answering one question following the 
reading of a response: Why did these people say this response? Sometimes they won't say, 
other times that's all they will say. Reasons can be identified by the use of "for" and "for 
my . 

Not all reasons are contained in these codes. Look to the action codes for responses that are 
worded like reasons. Primarily the reason codes were developed because these verbatim 
"reasons" frequently appeared. Most reason responses can be coded through verbatim coding 
only. 

Rl to put (NOT GET) a book, film, video on reserve : Used mostly by professors who PUT 
books on reserve for their students. DO NOT use this code when a respondent is getting a 
book from reserve, use the getting object code 3A, for that. 

R2 for research: Use verbatim, if you don't see this word , don't code it. 

R3 for a paper, dissertation, thesis, report, project: Another verbatim code, when you see it 
or anything close to it, use it. Reread Coding Priorities Paragraph C again to remind yourself 
how to code research with project. You'll see this allot. 

R4 to make bibliography more complete: Use verbatim, if you don't see these words , don't 
code it. 

R5 stay current, catch-up, recent: Used by respondents who are staying current, seeing what 
latest information is out there or for students who want to catch-up on back work. 

R6 for leisure, pleasure, personal interest (not related to work), browsing: Verbatim as well 
as concept code. Respondents will say they're doing it for fun, leisure, relax, an interest not 
related to anything else should be coded here. If it's for a job, that's a personal interest - 
code here. Browsing is a verbatim, if you don't see it, don't code it. 

R7 for a class, studies: You'll see this in many forms, specifically they'll name the class. 
When a respondent mentions particular subjects, i.e. Ancient Roman Architecture or 
Accounting, these should be coded "for a class". Many times a paper they'll be working will 
be for this reason 

R8 to point to another source of information: A concept code you won't sec frequently. 
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Some will say "This information lead me to another source which..." clearly, it pointed them 
to other sources of information. 

R9 for a student(s): Used by professors, you won't see it much. 

RIO Staff Performance 

RI0.A Positive (people were nice, helpful,good at job, efficient did what 
they were supposed to do) 

R10.B Negative (was not notified, had difficulties, unskilled clerks) 
These codes will be present anytime a respondent talks about the Library Staff. 

Rll Allows access to inaccessible materials, go and get later, it's available: A seemingly 
strange code but people say it. Close to verbatim responses here. 

Rll Didn't' want to travel, want to access from home: Use verbatim, if you don't see these 
words , don't code it. 

RI3 freedom from worry, reduces stress, makes teaching easier: Use verbatim, if you don't 
see these words , don't code it. 

R14 It's part of my job: Not used much, put people do use the Library in the fulfillment of 
their job related duties or one could interpret it that way. 

R15 Physical environment 

R15.A could be better 

R15.B not big enough 

R15.C material not organized well 
You'll see this code exclusively on question 22. Almost verbatim. This code represents 
peoples' responses about what they think of the physical library environment. 

R16 Procedure or library rules: 
R16.A Fines too high 

RI6.B Check out books for longer periods of time, time limits 

R16.C Library doesn't accept delivers 
Again, You'll see this code exclusively on question 22. Almost verbatim. This code 
represents peoples' responses about what they think of the library's procedures. 

RI7 Equipment performance 

R17.A Worked well, user friendly 

R17.B Didn't work well, not designed well, not user friendly 
This code will be seen in question 22 and question 10. Use it for respondents' comments on 
system and equipment performance. Good comments get R17.A, bad comments get R17.B. 
Don't confuse this code with the good and bad convenience codes; 3D and 4C. 
[These are illustrative. Similar detailed instructions were used in applying the other 
components of the empirical taxonomy.] 
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1. Assignment of Questions to Coding schemes 

The following is a list of all of the questions that have been coded according to each of the 
open-ended coding schemes described in this report. The labels are, for example W1.2Q17= 
"Wave 1.2 Question 17". 



Code Class: Done — Coded What Would Have Done 



W1Q17 [16. If service name were not available at this library, is there 

something else you could have done to get. the same benefit?]. 
What would you have done? 

W1.2Q12A If service name were not available at this library, is there something 
else you could have done to get this same benefit? 

Code Class: Getout - Coded by What you Getout 

W1Q6 Why did you use service name today? 

What did you get out of using service name today? 



W1Q8 
W1Q10 



W1Q22 



W2P1 

W2P3a 
W2P3b 

W2A1 



[9. Using a scale from 1 to 7 where 1 means not at all successful and 7 
means very successful, how successful were you in getting what 
you needed from service name?] . What would you say is the 
reason you feel that way? 

[21. If 100 represents the best possible experience with service name 
that you could imagine, and 0 represents the worst, what score 
would you give for the way it actually turned out?]. Why did 
you choose that score? 

Thinking about the last time you used service name , what was the 
actual project or work that brought you to use it? 

What did you get out of using service name on that occasion? 

What would you say was the value or benefit or impact of that for you 
and for your work? 

What is the actual project or work that brought you to use service name 
today? 



W2C1A 



What did you get out of using service name on that occasion? 



W2C1B What would you say was the value or benefit or impact of that for you 
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and for your work? 
Wl .2Ql Why did you use service name today? 

W1.2Q3 What is the actual project or work that brought you to use this 

service today? 

W1.2Q4 What did you get out of using service name today? 



Code Class: Reason - Coded According to Reason 

W1.2Q7 [6. Using a scale from 1 to 7 where 1 means not at all confident 

and 7 means extremely confident, how confident are you that 
you can rely on the information you got from using service 
name today?]. What would you say is the reason you feel that 
way? 

W2P5a [5. Using a scale from 1 to 7 where 1 means not all confident and 7 

means extremely confident, how confident are you that you can 
rely on the information you got from using service name on that 
occasion?]. What would you say is the reason you feel that 
way? 

W2A5 [4. Using a scale from 1 to 7 where 1 means not at all confident and 7 

means extremely confident, how confident are you that you can 
rely on the information you got from using service name 
today?]. What would you say is the reason you feel that way? 

Code Class: Choose -- Coded on Choose Scheme 

W1.2Q17 [16. If 100 represents the best possible experience with service name 
that you could imagine, and 0 represents the worst, what score 
would you give for the way it actually turned out today?]. Why 
did you choose that score? 



Code Class: Way -- Coded According to Way 

W1.2Q5a 1 5. On a scale of 1 to 7 with 1 being not at all helpful and 7 being 
very helpful, how helpful was what you got to your actual 
project or work?]. In what way was this helpful to your project 
or work? 
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W2C2? [2. On a scale of 1 to 7 with 1 being not at all helpful and 7 being 

very helpful, how helpful was what you go to your actual project 
or work?]. In what way was this helpful to your project or 
work? 

W2P4a [4. On a scale of 1 to 7 with 1 being not at all helpful and 7 being 

very helpful, how helpful was what you got to your actual 
project or work?]. In what way was this helpful to your project 
or work? 



Code Class: Hurt -- Coded by how it would hurt 

W1.2Q5b [5. On a scale of 1 to 7 with 1 being not at all helpful and 7 being 
very helpful, how helpful was what you got to your actual 
project or work?]. If you had not gotten what you did, how 
would it have hurt your project or work? 

W1.2Q5c In what way was it not helpful to your project or work? 

W1.2Q5e How did it hurt your project or work? 

W2C2Qb [2. On a scale of 1 to 7 with 1 being not at all helpful and 7 being 
very helpful, how helpful was what you got to your actual 
project or work?]. If you had not gotten what you did, how 
would it have hurt your project or work? 

W2P4b [4. On a scale of 1 to 7 with 1 being not at all helpful and 7 being 

very helpful, how helpful was what you got to your actual 
project or work?]. If you had not gotten what you did, how 
would it have hurt your project or work? 

W2P4c In what way was it not helpful to your project or work? 

W2P4e How did it hurt your project or work? 

W2A3a [3. on a scale of 1 to 7 with 1 being not at all helpful and 7 being very 
helpful, how helpful was what you got to your actual project, or 
work?]. In what way was it not helpful to your project or work? 



W2A3c 



How did it hurt your project or work? 



APLAB-B%D\CLR\FINAL.REV\APPSA-B.51 Copyright (c) 1995 „ _ March 21, 1995 P. 64 

0 'j ;i 
i.) 



> 



Rutgers Alexandria Project Lab Cost Benefit Study 1994 
Paul B. Kantor and Tefko Saracevic, Principal Investigators 

Appendix B 



B3. Link of empirical codes used in this study (the Empirical Taxonomy) to the conceptual 
taxonomy(Derived Taxnomy) proposed in Chapter 4 of the principal report. Except for the 
code class "Getout", present codes are indicated by Question number and instrument, as 
"Q17t2A" means "Code 2A of the scheme used to code Question 17 on the 1.2 ("t" for "two") 
instrument. 



Present 
Class "C 
1 

1A 
IB 
1C 
2 

2A 
2B 
3 

3A 
3A 
3B 
3B 
3C 
3D 
3E 
3F 
3F 
3F 
3F 
3F 
3F 
4 

4A 

4A 

4B 

4B 

4B 

4C 

4D 

4D 

4D 

4D 

4D 

4D 

RT 

RT1 

RT2 

RT2 

RT2 

RT3 

RT4 

RT5 

RT6 

RT6 

RT7 

RT8 

RT9 

RS 

RSI 

RS2 

RP 

RP1 

RP2 

RP3 

RE 

RL 

RL1 





Proposed 


RL2 


B.3 


2 . 2 


017 




B . i. 






;out 1 




RL3 


B . 3 


2 . 3 


017 


1 


B. 


4 


10 


t\ . 






RR 


B . 3 






2 


B. 


4 


9 


A . 


2 


x . o 


RR1 


B . 3 


1 . 3 


m 7 


2A 


B. 


4 


9 


A . 


2 


x . o 


RR2 


R 3 


1 9 
-L • Z 


m 7 


OR 


B. 


4 


Q 


A . 


2 


1 . 3 


RR3 


NOT 


USED 


m 7 


2C 


B. 


4 


Q 




3 




RP 


B . 3 


3 


m 7 


2D 


B . 


4 


9 


A . 


3 




RP 


B . 3 


4 


m 7 


2E 


B. 


4 


9 






~> . O 


RPE 


B . 3 


4 


m 7 


^ r 


B. 


4 


Q 




3 




RPE1 


B . 3 


4 . 3 


m 7 


2G 


B 


4 


9 






1 


t\£T EjZ 


R i. 


4 3 


m 7 
y± ' 


■3 


B 


4 


1 




•5 


1 1 
-L . X 




B . 3 


. 3 


m 7 




B 


4 


1 
j. 




3 




RPS1 


B . 3 


3 1 

. O . X 


m 7 


"3 R 


B 


4 






3 


J • Z 


RPS1 


R i. 


3 0 

. O . Z 


m 7 


/I 
** 


B 


4 


z 


c 


5 


1 


RPS1 


B . 3 


.3.3 


m 7 




B 


4 


Z 


B 


2 


1 
J. 


RPS2 


B . 3 


.3.1 


m 7 


4B 


B 


4 


2 


C 


6 


2 


RPS2 


B . 3 


.3.2 


m 7 


/in 


B 


4 


z 


C 


4 




RPS2 


B.3 


.3.3 


Q17 


5 


B 


4 


3 


C 


4 


1 








Q17 


5A 


B 


4 


3 


C 


3 










Q17 


. 5A1 


B 


4 


3 


C 


3 


1 








Q17 


. 5A2 


B 


4 


3 


C 


3 


.2 








Q17 


. 5B 


B 


4 


3 


C 


4 


.5 








Q17 


. 5C 


B 


4 


3 


A 


3 










Q17 


.6 


B 


4 


6 


B 


1 










Q17 


. 6A 


B 


4 


.6 


B 


1 


.1 








Q17 


. 6B 


B 


4 


.6 


C 


5 


.1 








Q17 


.7 


NOT 


.USED 


C 


5 


.2 


















C 


5 


# 3 


















B 


2 


.1 


















C 


4 




















C 


.4 


.1 


















C 


.3 




















C 


.3 


.1 


















B 


.1 


.2 


















B 


.1 


.3 


















A 


.1 




















A 


.1 


.1 


















A 


.1 


.2 


















A 


.1 


.3 


















A 


.1 


.4 


















A 


.1 


.6 



















N0T_USED 
N0T_USED 
A.l .7 
A. 3 .3 .1 
A. 2. 1.2 
A. 3 .2 .2 
N0T_USED 
A. 1.15 
A.l .15 
A.l .15 
A. 2 

A. 2. 2.1 

A. 2 .2 .2 
N0T.USED 

B. 3 .2 
B.3 .2 .2 
B.3 .2 .2 



9 

I :RIC 
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Q17t 


LI 


B . 3 


O 1 

J . 1 




Q17t 


L2 


B . J 


^ . Z 


Q 5 A . H 


Q17 1 


L3 


B . 3 


3 . 2 


Q5A . HI 


Q17t 


L4 


B . 3 


3 . 1 


Q5A . H2 


Q17t 


Ql 


C . 5 


1 


Q5A .H3 


Q17t 


Q2 


C . 5 


2 


Q5A.H4 


Q17t 


Q3 


C . 6 


2 


Q5A . Gl 


QI / 1 


Q4 


B . Z 


1 


y DA . 


Q17t 


PR1 


C.l 


4 


Q5A.G3 


Q17t 


PR 2 


NOT 


USED 


Q5A.G4 


Q17t 


PI 


B.3 


3.2 


Q5A.G5 


Q17t 


P2 


B.3 


3.2 


Q5A.G6 


Q17t 


Gl 


C.4 


1 


Q5A.T1 


Q17t 


G2 


C.4 


1 


Q5A.T2 


Q17t 


G3 


C.4 


1 


Q5A.M1 


Q17t 


El 


B.3 


4.1 


Q5A.N 


Q17t 


.El 


B.3 


4.3 


Q5A.D 


Q17t 


.E2 


B.3 


.4.1 




Q17t 


.E2 


B.3 


.4.3 




Q17t 


.E3 


NOT 


USED 











xxxx 


XXX XX 


C 


3 




QR 


Nl 


C .4 . 6 


C 


3 


1 


QR 


SI 


C . 4 . 6 


C 


3 


1 


QR 


T 1 


C . 4 . 1 


C 


3 


1 


QR 


12 


C.4 .3 


C 


3 


4 


QR 


13 


B.4 .1 


C 


3 


2 


QR 


14 


B.4 .3 


C 


1 


4 


QR 


Ml 


B.6 . 5 


C 


3 


1 


QR 


Tl 


C.5.2 


C 


4 


1 


QR 


PI 


C.4 .4 


C 


4 


1 


QR 


P2 


C.4 .6 


C 


3 


3 


QR 


P3 


C.1.6 


C 


5 


1 


QR 


Gl 


B.4 .2 


C 


5 


2 


QR 


Gl 


B.4 .3 


C 


6 


.2 


QR 


H2 


C . 4 . 6 


C 


4 


.3 








C 


.4 


.4 
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Q7 . 


11 


NOT_USED 


Q12A . 


PI 


E . 


4 


1 


Q7 . 


12 


NOT_USED 


Q12A . 


P2 


B . 


4 


3 


Q7 


13 


NOT_USED 


Q12A . 


P3 


B 


4 


3 


Q7 


14 


NOT_USED 


Q12A . 


P5 


B . 


4 


6 


Q7 


15 


NOT_USED 


Q12A . 


P6 


C 


6 


4 


Q7 


16 


C . 1 . 4 


Q12A 


Ml 


B 


4 


9 


Q7 


17 


B.l .3 


Q12A 


M2 


B 


4 


9 


Q7 


PI 


C .4 .1 


Q12A 


M3 


B 


4 


10 


Q7 


P2 


C.4 .1 


Q12A 


11 


B 


4 


1 


Q7 


P3 


C . 4 . 3 


Q12A 


12 


B 


4 


1 


Q7 


P4 


C .4 .4 


yl2A 


13 


B 


4 


1 


Q7 


P5 


C . 4 . 1 


Q12A 


14 


B 


4 


1 


Q7 


P6 


C.4 .4 


Q12A 


15 


B 


4 


6 


Q7 


E2 


B.3 .3 .2 


Q12A 


16 


B 


4 


2 


Q7 


E3 


B.3 .3 .1 


Q12A 


17 


B 


4 


2 


Q7 


E5 


B.3 .3 .2 


Q12A 


18 


B 


4 


.1 


Q7 


E6 


B.3 .3 .1 


Q12A 


19 


B 


4 


.1 


Q7 


.S2 


C.4 .4 


Q12A 


110 


B 


4 


. 1 


Q7 


• S3 


NOT USED 


Q12A 


Rl 


B 


4 


.9 


Q7 


.S4 


B.l .4 


Q12A 


T3 


B 


2 


.3 


Q7 


.S4 


B.2 .1 


Q12A 


T4 


B 


2 


.3 


Q7 


.S5 


C.5.1 


Q12A 


LI 


B 


4 


.8 


Q7 


.S6 


C.4 .5 












Q7 


.S7 


A. 2 .1 .3 












Q7 


.ss 


C.4 .4 












Q7 


.S10 


C.4 .1 












Q7 


. SIO 


C.3 .1 
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The following reverse table is provide to assist in tracking the proposed codes to the 
codes used in this study. Here the entries are listed m order of their proposed 
codes, with multiple entries for proposed codes that map to more than one of the codes 
used in this study. 









Q17t.Ll 


B. 


3 


3 


1 


Q12A.P1 


B.4 . 


1 


RT 


A.l 




Q17t .L4 


B 


3 


3 


1 


Q12A. 11 


B.4 . 


1 


RT1 


A.l . 


1 


Q7.E3 


B 


3 


3 


1 


Q12A. 12 


B.4 . 


1 


RS 


A.l. 


15 


Q7.E6 


B 


3 


3 


1 


Q12A.I3 


B.4 . 


1 


RSI 


A.l . 


15 


RPS1 


B 


3 


3 


2 


Q12A.I4 


B.4 . 


1 


RS2 


A.l . 


15 


RPS2 


B 


3 


3 


2 


Q12A. 18 


B.4 . 


1 


RT2 


A.l 


2 


Q17t.L2 


B 


3 


3 


2 


Q12A. 19 


B.4 . 


1 


RT2 


A.l 


3 


Q17t .L3 


B 


3 


3 


2 


Q12A.I10 


B.4 . 


1 


RT2 


A.l 


4 


Q17t . PI 


B 


3 


3 


2 


Q17 .1 


B.4 . 


10 


RT3 


A.l 


6 


Q17t.P2 


B 


3 


3 


2 


Q12A.M3 


B.4 . 


10 


RT6 


A.l 


7 


Q7.E2 


B 


3 


3 


2 


P17 .4 


B.4 . 


2 


RP 


A. 2 




Q7.E5 


B 


3 


3 


2 


Q17 .4A 


B.4 . 


2 


RT7 


A. 2 


1 .2 


RPS1 


B 


3 


3 


3 


Q17 .4B 


B.4 


2 


1 


A. 2 


1.3 


RPS2 


B 


3 


3 


3 


Q17 .4C 


B.4 


2 


1A 


A. 2 


1.3 


RP 


B 


3 


4 




QR.G1 


B.4 


2 


IB 


A. 2 


1.3 


Q17t .El 


B 


3 


.4 


1 


Q12A.I6 


B.4 


2 


1C 


A. 2 


1 .3 


Q17t.E2 


B 


3 


.4 


1 


Q12A. 17 


B.4 


2 


Q7.S7 


A. 2 


1.3 


RPE1 


B 


3 


.4 


.3 


Q17 .5 


B.4 


3 


RP1 


A. 2 


2.1 


RPE2 


B 


3 


.4 


.3 


Q17 . 5A 


B.4 


3 


RP2 


A. 2 


2.1 


Q17t.El 


B 


3 


.4 


.3 


Q17 . 5A1 


B.4 


3 


3 


A. 3 




Q17t.E2 


B 


.3 


.4 


.3 


Q17 . 5A2 


B.4 


3 


4 


A. 3 




Q17 


B 


.4 






Q17 . 5B 


B.4 


3 


3A 


A. 3 


1 


Q17.3 


B 


.4 


. 1 




Q17 .5C 


B.4 


3 


3A 


A. 3 


1.1 


Q17.3A 


B 


.4 


. 1 




QR.I4 


B.4 


3 


3B 


A. 3 


2 . 1 


Q17 .3B 


B 


.4 


. 1 




QR.G1 


B.4 


3 


RT8 


A. 3 


2 .2 


QR.I3 


B 


.4 


. 1 




Q12A. P2 


B.4 


3 


2 


A. 3 


3 












Q12A.P3 


B.4 


3 


2A 


A. 3 


3 












Q17 .6 


B.4 


6 


RT6 


A. 3 


.3 .1 












Q17 . 6A 


B.4 


6 


3B 


A. 3 


.3 .2 












Q17 . 6B 


B.4 


6 


2B 


A. 3 


.3 .6 












012A.P5 


B.4 


6 


4A 


B.l 














Q12A.I5 


B.4 


6 


4A 


B.l 


.1 












Q12A.L1 


B.4 


8 


4D 


B.l 


.2 












Q17 .2 


B.4 


9 


4D 


B.l 


.3 












Q17 .2A 


B.4 


9 


Q7 . 17 


B.l 


.3 












Q17 .2B 


B.4 


.9 


Q7.S4 


B.l 


.4 












Q17 .2C 


B.4 


.S 


3D 


B.2 


.1 












Q17 .2D 


B.4 


.9 


4C 


B.2 


.1 












Q17 .2E 


B.4 


.9 


Q17t.Q4 


B.2 


.1 












Q17 .2F 


B.4 


.9 


Q7.S4 


B.2 


.1 












Q17 .2G 


B.4 


.9 


Q12A.T3 


B.2 


.3 












Q12A.M1 


B.4 


.9 


Q12A.T4 


B.2 


.3 












Q12A.M2 


B.4 


.9 


RR 


B.3 


.1 












Q12A.R1 


B.4 


.9 


RR2 


B.3 


.1.2 












QR.M1 


B.6 


.5 


RR1 


B.3 


.1.3 












Q17t.PRl 


C.l 


.4 


RE 


B . 3 


.2 


















RL 


B.3 


.2 .2 












Q5A.G2 


C.l 


.4 


RL1 


B.3 


.2 .2 












Q7 .16 


C.l 


.4 


RL2 


B.3 


.2.2 












QR. P3 


C.l 


.6 


RL3 


B.3 


.2 .3 












3F 


C.3 




RP 


B.3 


.3 












4D 


C.3 




RPS 


B.3 


.3 












Q5A.H 


C.3 




RPS1 


B.3 


.3 .1 












3F 


C.3 


.1 


RPS2 


B.3 


.3.1 












4D 

Q5A.H1 
Q5A.H2 
Q5A.H3 
Q5A.G3 
Q7.S10 
3F 

Q5A.G1 
Q5A.G6 
Q5A.H4 
3F 


C.3 
C.3 
C.3 
C.3 
C.3 
C.3 
C.3 
C.3 
C.3 
C.3 
C.4 


. 1 
. 1 
.1 
.1 
.1 
. 1 
.2 
.2 
.3 
.4 
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4D 
3F 

Q17t.Gl 
Q17t.G2 
Q17t.G3 
Q5A.G4 
Q5A.G5 
QR. II 
Q7 . PI 
Q7 . P2 
Q7 . P5 
Q7 . SIC 
4D 

Q5A.N 
QR. 12 
Q7 .P3 
Q5A.D 
QR. PI 
Q7 . P4 
Q7 . P6 
Q7 . S2 
Q7 .S8 
3F 

Q7.S6 

QR.N1 

QR.S1 

QR. P2 

QR.H2 

3C 

4B 

Q17t.Ql 
Q5A.T1 
Q7.S5 
4B 

Q17t.Q2 

Q5A.T2 

QR.T1 

4B 

3E 

Q17t.Q3 

Q5A.M1 

Q12A. P6 

RT4 

RT5 

RT9 

RP3 

RR3 

Q17 .7 

Q17t.PR2 

Q17t.E3 

Q7 . II 

Q7 . 12 

Q7.I3 

Q7.I4 

Q7.I5 

Q7.S3 



C .4 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .1 
C.4 .3 
C.4 .3 
C.4 .3 
C.4 .4 
C.4 .4 
C.4. 4 
C.4 .4 
C.4 .4 
C.4 .4 
C.4. 5 
C.4. 5 
C.4 .6 
C.4. 6 
C.4. 6 
C.4 .6 
C.5.1 
C.5.1 
C.5.1 
C.5.1 
C.5.1 
C.5.2 
C. 5.2 
C.5.2 
C.5.2 
C.5.3 
C.6.2 
C.6.2 
C.6.2 
C.6.4 
NOT L JED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
NOT USED 
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|Note:A list of all appendices appears at the end of the file: finalbig.psl51] 

APPENDIX C 



Open Ended Coding Result Tables 
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We present here the full results for each coding scheme. In the body of 
the report (Chapter 4) we presented only those responses which occured 
represented at least 10% of the responses. Note that because schemes were 
evolving during the study, not all schemes were applied to 528 cases. Some were 
used only in Wave 1, or only in later waves, respectively. Please refer ot 
Appendix B for a complete list of the questions which wre coded according to ea=h 
of the schemes shown here. 



Group $CHOOSE Coded on Choose Scheme 









Pet 


of 


Pet of 


Category label 


Code 


Count 


Responses 


Cases 


Total 


responses > 


109 


100 


0 


143 .4 


Got averything I wanted 


51 


22 


20 


2 


28.9 


Save time 


21 


14 


12 


8 


18.4 


Helpful finding 


12 


13 


11 


9 


17 . 1 


Could have been more 


53 


11 


10 


1 


14.5 


Got most 


52 


10 


9 


2 


13 . 2 


More easy, convenient 


24 


10 


9 


2 


13 . 2 


Personal reason 


30 


5 


4 


6 


6 . 6 


I'm still learning 


63 


4 


3 


7 


5.3 


Equipment doesn't work well 


62 


4 


3 


7 


5.3 


Gave me a direction 


31 


4 


3 


7 


5.3 


Too much time 


22 


3 


2 


8 


3 . 9 


Look it up myself 


32 


2 


1 


8 


2 . 6 


Saved money 


23 


2 


1 


8 


2 . 6 


Equipment does work well 


61 


1 




9 


1.3 


Bad physical environment 


42 


1 




9 


1 . 3 


Not in my area of specialization 


14 


1 




9 


1 . 3 


Not helpful finding 


13 


1 




9 


1.3 


Problems getting my point 


11 


1 




.9 


1.3 



452 missing cases; 76 valid cases 
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Group $DONE Coded What Would Have Done 









Pet 


of 


Pet of 


Category label 


Code 


Count 


Responses 


Cases 


Total responses > 


317 


100 


0 


131 . 0 


Another library 


4 


44 


13 


o 


18 . 2 


Another university 


42 


38 


12 


o 


15 . 7 


Another place 


5 


25 


7 


9 


10 . 3 


A public library 


41 


21 


6 


6 


8.7 


Interlibrary loan or recall 


. 31 


19 


6 


0 


7.9 


Card catalog 


26 


19 


6 


0 


7 . 9 


Buy items 


511 


18 


5 


7 


7 .4 


Check book" 3 "Otheir library services" 


27 


17 


5 


4 


7 . 0 


Indexes 


21 


16 


5 


0 


6.6 




2 


15 


4 


7 


6.2 


Librarian 


32 


10 


3 


2 


4.1 


Abstracts 


24 


10 


3 


2 


4.1 


On-line (electronic) services 


23 


8 


2 


5 


3.3 


Prof ess or /adviser /teacher 


62 


7 


2 


2 


2 . 9 


Do nothing 


1 


7 


2 


2 


2.9 


Friend 


61 


6 


1 


9 


2.5 


My work 


53 


6 


1 


9 


2 . 5 


Another computer facility 


52 


6 


1 


9 


2 . 5 


People 


6 


6 


1 


. 9 


2.5 


Bookstore/record store 


51 


5 


1 


. 6 


2.1 


Rent items 


512 


3 




.9 


1.2 


Randomly look 


22 


3 




. 9 


1.2 




3 


3 




. 9 


1.2 


Readers guide 


25 


2 




. 6 


.8 


Miscellaneous 


7 


2 




. 6 


. 8 


Checking computer system 


43 


1 




. 3 


.4 



286 missing cases; 242 valid cases 
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Group $GETOUT Coded by What you Getout 









Pet 


of 


Pet of 


Category label 


Code 


Count 


Responses 


Cases 




Total responses > 


3497 


100 


0 


662 . 3 


Get physical Object 


31 


459 


13 


1 


86 . 9 


Information, knowledge 


32 


407 


11 


6 


77 . 1 


For a paper 


52 


383 


11 


0 


72 . 5 


Work 


2 


271 


7 


7 


51 . 3 


Orient 


1 


265 


7 


6 


50 . 2 


Task completed 


36 


212 


6 


1 


40 . 2 


For research 


51 


192 


5 


5 


36.4 


For a class 


56 


174 


5 


0 


33 . 0 


Save time 


33 


168 


4 


8 


31.8 


Convenience, efficiency 


34 


146 


4 


2 


27 .7 


Didn't fulfill goal 


44 


119 


3 


4 


22.5 


Not available 


41 


110 


3 


1 


20.8 


Positive (People were nice) 


941 


53 


1 


5 


10 . 0 


For personal reasons 


7 


45 


1 


3 


8 . 5 


Go and get later 


59 


37 


1 


1 


7 . 0 


Didn't work well 


932 


31 




9 


5 . 9 


Locations of items 


11 


31 




9 


5 . 9 


Save money 


35 


30 




9 


5.7 


Learn to use resources 


13 


30 




9 


5.7 


Part of my job 


55 


29 




8 


5 . 5 


Stay current 


57 


28 




8 


5 . 3 


Took too long 


42 


28 




. 8 


5.3 


Didn't want to travel 


73 


25 




.7 


4.7 


For a student 


62 


24 




.7 


4.5 


Point to another source 


58 


23 




. 7 


4.4 


Not convenient 


43 


20 




. 6 


3 . 8 


Time limits 


822 


16 




. 5 


3 . 0 


For leisure 


71 


14 




. 4 


2.7 


Negative (was not notified) 


942 


12 




. 3 


2 . 3 


Place for work 


21 


12 




. 3 


2 . 3 


User friendly 


931 


11 




.3 


2 . 1 


Make bibliography 


53 


9 




.3 


1.7 


Availability of resour~.es 


12 


9 




. 3 


1.7 


Reduces stress 


72 


8 




. 2 


1 . 5 


For a professor 


61 


8 




. 2 


1 . 5 


Put on reserve 


54 


8 




.2 


1 . 5 


For task accomplishment 


5 


8 




. 2 


1 . 5 


Could be better layout 


811 


6 




. 2 


1 . 1 


Use computer 


22 


6 




. 2 


1 . 1 


Fines too high 


821 


5 




. 1 


. 9 




70 


4 




. 1 


. 8 




50 


4 




. 1 


. 8 


For someone 


6 


4 




. 1 


. 8 


Get 


3 


3 




. 1 


. 6 


Doesn't accept delivers 


823 


2 




. 1 


.4 


Not well organized 


813 


2 




. 1 


. 4 


Procedures /rules 


82 


2 




. 1 


. 4 


Not big enough layout 


812 


1 




. 0 


. 2 


Performance 


83 


1 




. 0 


. 2 


Library environment 


8 


1 




. 0 


.2 
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Did noc get 

0 missing cases; 528 valid cases 
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Group $HURT Coded by how it would hurt 









Pet 


of 


Pet of 


Category label 


Code 


Count 


Responses 


Cases 


Total responses > 


230 


100 


0 


124.3 


Not able to complete 


10 


66 


28 


7 


35.7 


Taken more time 


50 


53 


23 


0 


28. 6 


Gotten less information 


31 


26 


11 


3 


14.1 


Other information source 


34 


20 


8 


7 


10 . 8 


Gone physically somewhere else 


70 


15 


6 


5 


8.1 


Not gotten information 


32 


7 


3 


0 


3 . 8 


Hurt a great deal 


62 


6 


2 


6 


3.2 


Not sure 


61 


6 


2 


6 


3.2 


It wasn't important 


80 


5 


2 


2 


2.7 


Personal reason 


60 


5 


2 


2 


2.7 


Contact someone or figure it out 


20 


5 


2 


2 


2.7 


Tl-(time) Would have taken more time 


51 


4 


1 


1 


2 .2 


Losing money 


40 


4 


1 


1 


2.2 


Same resources in different form 


33 


4 


1 


.1 


2.2 


Not knowing where to start 


63 


3 


1 


.3 


1.6 




53 


1 




.4 


. 5 



343 missing cases; 185 valid cases 
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Group $REASON Coded according to reason 

Pet of Pet of 



Category label 


Code 


Count 


Responses 


Cases 


Total 


responses > 


220 


100 . 


0 


126 . 


4 


Confident in service 


42 




31 


14 


1 


17 


8 


Used the service in the past 


43 


29 


13 


2 


16 


7 


Not complete, comprehensive, useful 


40 


22 


10 


0 


12 


6 


Helpful 


32 


17 


7 


7 


9 


8 


Double check 


15 


12 


5 


5 


6 


9 


I have trust 


48 


10 


4 


5 


5 


7 


Not sure 


24 


10 


4 


5 


5 


7 


Finding everything 


21 


10 


4 


5 


5 


7 


A lot of information 


46 


9 


4 


1 


5 


2 


Personal reason 


20 


9 


4 


1 


5 


2 


To other info 


16 


9 


4 


1 


5 


2 


Easily accessible 


44 


8 


3 


6 


4 


6 


Specialized to my topic 


12 


8 


3 


6 


4 


6 


Saves time 


45 


7 


3 




4 


0 


Confidence up to a certain date 


26 


6 


2 


7 


3 


4 


Not up-to-date 


17 


4 


1 


8 


2 


3 


Learn to use it 


47 


3 


1 


4 


1 


7 


Experience tells me 


25 


3 


1 


4 


1 


7 


Wasn't successful 


23 


3 


1 


4 


1 


7 


Omitted 


14 


3 


1 


4 


1 


.7 


Finding most. 


22 


2 




9 


1 


. 1 


Not comfortable 


36 


1 




. 5 




. 6 


Not helpful 


35 


1 




. 5 




. 6 


Comfortable 


33 


1 




. 5 




. 6 


General 


13 


1 




. 5 




. 6 


Fac'.s not opinions 


11 


1 




. 5 




. 6 


354 missing cases; 174 valid cases 













APLAB-B%D\CLR\FINAL.REV\APPSC-H.51 Copyright (c) 1995 



March 21, 1995 P. 78 



Rutgers Alexandria Project Lab Cost Benefit Study 1994 
Paul B. Kantor and Tefko Saracevic, Principal Investigators 

Appendix C 



Group $WAY Coded according to way 









Pet 


of 


Pet of 


Category label 


Code 


Count 


Responses 


Cases 




Total responses > 


203 


100 


0 


111 . 5 


Information/references 


24 


58 


28 


6 


31 . 9 


Helped me start 


14 


27 


13 


3 


14.8 


Complete or help my work 


23 


23 


11 


3 


12 . 6 


Ideas, insight, perspective 


22 


22 


10 


8 


12 . 1 


How to find information 


12 


12 


5 


9 


6.6 


Saved time 


31 


11 


5 


4 


6.0 


Helped me 


10 


11 


5 


4 


6.0 


Up-to-date information 


25 


10 


4 


9 


5.5 


Find information 


13 


9 


4 


4 


4.9 


Don't know anything 


60 


7 


3 


4 


3.8 


Access to people 


26 


4 


2 


0 


2.2 


Did not find anything 


50 


3 


1 


5 


1 . 6 


Save -money 


40 


2 


1 


0 


1 . 1 


How to find 


11 


2 


1 


0 


1 . 1 


A better grade 


21 


1 




5 


. 5 


Gave me something 


20 


1 




5 


.5 



346 missing cases; 182 valid cases 
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APPENDIX D 



Cost Data Collection Forms 



Dl. Cover memo 

D2. Extracts from Tantalus Inc FUNCOST Manual 
D3. White Forms: Personnel and Other Direct Costs 
D4. Yellow Form. Estimate of Service Rendered 
D5. Return Instructions 
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LETTER TO LIERARY DIRECTORS AND LIBRARIANS PARTICIPATING IN THE STUDY 



Dear 

We are entering the last lap of the data collection for our project on measuring the costs and beneficial impacts 
of library functions. At this point everyone else on the team is tied up with analysis of the interview data, so I 
will be coordinating the cost part of the study myssll. 

In a nutshell, we want to estimate how much it costs (over the whole year) to provide each of the specific 
services studied at your library. We will divide that by the total number of times the service was used, to get an 
average unit cost for the service. 

In an ideal world (from an economist's point of view) you have both of these figures in some recent report, and 
can just send them to us directly. Because the world is usually not ideal, I am enclosing some forms and 
guidelines which may be helpful in estimating those numbers. 

I recommend that you delegate this task to someone who has a good understanding of what goes on in your 
library, and has access to salary figures. That person may in turn add one more colleague, to conduct brief 
interviews with staff obtaining estimates of their time allocated to the specific service. I strongly suggest that the 
responsibility not be further distributed, as it is hard to ensure that these guidelines will be interpreted 
consistently. 

Although I am traveling off and on for the next few weeks, I am promptly accessible via email as 
kantor@zodiac.rutgers.edu, and will try to answer any questions within 24 hours. 

The data analysis is coming together very nicely. We have completed the statistical analysis of Wave I, and are 
well along in the content analysis of that data. As several libraries have requested, we are preparing files * 
containing the transcribed texts of a!! responses at your library, arranged by question, and noi by respondent. 
We judge that this protects the privacy of the respondents in accordance with our human subjects regulations. 

Sincerely, 
Paul B. Kantor, 
Professor and Director 

PBK/wp End: Cost Data Collection Manual and Forms, 
cc: 
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TO: The person(s) collecting cost data 
FROM: Paul Kantor 

I have enclosed a few pages lifted from an earlier manual on cost analysis. Since they arise 
out of context they may seem to be a little obscure, and I'll try to provide that context here. 

Costs are made up of labor costs and other direct costs. When a person (or a piece of 
equipment, or a kind of supplies) is 100% dedicated to some specific service, there is no 
problem. We count 100% of salary, or 100% of the direct cost, as a cost of that service. 

But when a person, a supply or a piece of equipment is shared by several services, we must 
do a little more work. For the supplies or equipment we ask you to estimate an allocation 
directly to the service of interest. An allocation will have a unit (in which the allocation is 
measured), a basis (the total number of those units) and a share (the number assigned to the 
particular service. 

For example, if a central computer supports 210 terminals, of which 103 are used for the 
OLPAC, the analysis is: 

Service= Computer 
Cost/yr= 400,000 
Unit = terminals 
Basis 210. 
Share = 103. 

If you want to complete the calculation and just send us the result ( $400,000 x 103/210 = 
196,191 ) I won't complain, but we are set up to do all the calculation very easily. To 
simplify things we provide a form (you may make as many copies as you need, to cover 
various direct costs) in which all of the data shown here will fit onto a single row. 

For labor costs our guiding principle is the same, but the procedure must be a little more 
complicated. For people who work 100% on the service being costed, their whole salaries are 
entered. But many people who work in libraries divide their time among tasks and services. 
We deal with this by a method called "guided introspection". This is much less burdensome 
than keeping logs or diaries, or doing observational work-s-.mpling studies. 

The key to guided introspection is to ask each person how much time goes into the service at 
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hand. Ai noted on the sheets, we can deal with answers given as a percentage, as hours per 
day, or as hours per week. We have found, in numerous studies, that this method gives 
sufficient accuracy for purposes of functional cost analysis. 

HOWEVER, there is one pitfall to avoid. When a person is interviewed about a specific 
service (say, materials delivery) that service assumes increased importance in that person's 
mind, leading to potential inflation of the cost of the service under study. To control for this 
we try always to situate the service of interest in the context of other services on which that 
person works. 

If you know the array of services on which the people to be interviewed work, you may 
prepare the data form in advance. This works especially well if there are only a few "other 
services" to be considered. If the people to be interviewed work on a host of different 
services you will do better to simply use a new sheet for each person, and begin by asking: 
"what are the different services you work on?". You might also suggest a few, and be sure to 
include the service of interest. Next, ask the respondent which service or activity takes up the 
most time, and how much it takes up. Then continue in decreasing order of time spent on 
services until you have covered them all. 

We have allowed special columns for tallying "professional development" which is a part of 
the working load for professional librarians, and "administrative meetings" which play a 
greater or lesser role, according to the way in which your library is managed. 

The process may seem awkward at first, but becomes quite easy, and consumes less than 10 
minutes for each person interviewed. I consider it good practice to revisit each person about a 
week later, to ask whether the numbers you recorded still look reasonable, and to change 
them as needed. 

Adding salaries to the sheets should be done last, as these are frequently treated as 
confidential. Do not include fringe and overhead in the computed salary. 
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Extracts from the Tantalus Inc. FUNCOST Manual (Revised with Permission). How to Fill 
Out the Form for Allocation of Labor. 



We are looking for thoughtful but NOT totally documented estimates of how each 
staff member spends her/his time. We call this process "guided introspection." 

Different people think of their time in different ways. Some people think of their time 
in percentages, some people think of their time in hours per day, and others think of their 
time in hours per week. Any one of those schemes is all right. It doesn't matter which is 
used. 

In fact, we don't require that the hours add up right. Our examples below will show 
this. Person A in this example estimated that he spends 20 hours of a week on General and 
Administration and spends 15 hours per week on the Selection and Ordering of materials for 
the library. Then he added that he spends 10 hours per week at the reference desk. That 
adds up to 45 hours per week. We don't mind this. What we really want to know is that 
"reference is about half as much as the administration and that the selection and acquisition 
together are about 50% more than reference". We wouldn't even be alarmed if this person 
turned around and said: "Oh yes, I also spend 5 hours a wesk at the circulation desk." So it 
is NOT important to make these numbers add up to the actual number of hours in the 
working week. What is important is the ratios. 

Person B is the kind of person who thinks of her time in terms of hours per day rather 
than hours per week. So she reports that she spends 3 hours per day on On-Line Reference 
and another hour a day in what she thinks of as the care and feeding of Reference. (Care and 
feeding referred to was time spent studying the manuals and learning how to do a better job 
of on-line searching. The time doesn't go into any particular search but it helps to improve 
the quality of all searches.) This person knows also that she spends 2 hours on the reference 
desk. This adds up so far to 6 hours. Finally, she told us that she doesn't know whereMwhe 
rest of the time goes. We record only the six known hours. We know that this person works 
eight hours a day. We believe that we have the ratios about right here, and we're not going 
to institute a full audit, just to find the rest of the hours. 

Person C is one who thinks in terms of percentages, and has reported that he spends 
40% of his time on cataloging materials and 40% of his time on inter-library loan and 40% of 
his time taking care of circulation. Again we don't mind that these add up to more than 
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100%. What is important is that they are about equal. He could have put 1. 1, and 1 and it 
would have given us the same information. 

Imaginary Dialogue. 

We include here extracts from a manual that has been sent to client libraries and 
libraries participating in some of our cost studies. This is an imaginary dialogue that you 
might have with a member of your staff when you're filling out the data collection form for 
allocation of labor. 

Consider a person whose initials are RCB. You don't need to put the salary down and 
you might not want to have all the salaries listed when you are talking to different 
individuals. Probably the best way to begin is by saying, "Which of these things (ILL. etc.) 
do you spent most of your time on?" 

A typical response might be, "I spend most of my time dealing with reference." 

If you have that response, point to the reference column and say, "In terms of hours 
per week or hours per day, or percentage-wise, how much of your time would you say you 
spend at that?" 

Suppose the answer is, "Three hours per day." 

Put a "3" on the line under "Ord Ref," and continue by saying, "What is the activity 
that takes up the next biggest part of your time?" 

The response might be: "Well the next biggest thing is that I spend a lot of time 
deciding which material to order. That takes about 20% of my time." 

This is the kind of information we can't deal with. We can't compare three hours 
with 20% so you have to say, "It. terms of hours per day how many hours would you say it 
comes out to?" 

Suppose the answer is: "About two." 

Put a two on the line under "Sel." At this point RCB may say, "1 guess I spend more 
time on reference than I thought because I think I spend about twice as much time on 
reference, so can we change it to four?" 
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The answer is: "Yes, we certainly can." Simply erase the three and put in a four. 
After this it gets a little more difficult because RCB has now accounted for about 75% of his 
time, and the rest is likely to be unclear; but you continue. You say, 

"There are a number of other categories listed here. Which others are you involved 

in?" 

A common answer would be, "Well I do a little of practically everything." Then you 
might say, "Which one of those other things takes the most of your time?" 

The answer might be, "Inter-library loan borrowing, the business of tracking down the 
books we want to get, that's probably my next biggest job and I would say that takes four or 
five hours a week." 

At this point you have to stop and say, "We can't write four or five here because 
everything else is in hours per day." But you can help by saying, "Well five hours a week 
would be about one hour a day, Is it correct to put a one in here?" 

If RCB says "yes," you have accounted for what looks like seven hours out of a 
working day which may consist of seven and a half or eight hours. Often, at this point, there 
is nothing more that can be said. RCB may say "The rest of my time just seems to be split 
up among a lot of different things. It changes from day to day." At that point you go to the 
next member of the staff. 

Let's say that Francis is the next employee (initials FXS) Begin as before by saying 
to Francis, "What activity takes up the largest part of your time during a typical working 
week?" 

Suppose the answer is: "It has to be working in circulation. I spend more time at the 
circulation desk than I do at anything else." 

Then you have to say, "WelSw about how much time? Is it four hours a day?" 

Francis may respond, "Well you know the schedule, it varies from day to day but I 
would say that it takes up 60% of my time." 

That means that on this row we are going to be dealing in percents, so you would put 
a 60 on the line under circulation. (It is not necessary to put the "%" sign.) Then you 
proceed as before by saying, "What is the thing you spend the next largest amount of time 
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doing?" 

Francis may say, "Well, I spend a lot of my time supervising three other people on the 
staff. Where would that go?" 

Supervisory work belongs under general administration, unless you know that the 
people supervised are doing only one narrowly defined task. In that case, supervising them 
should be assigned to that same task. In a small or branch library usually people share 
functions, so the general administration column would be more appropriate. In a main 
university library the supervision may be specific to one service. 

Now you have to get from Francis some reasonable estimate of how much of this time 
goes into supervisory activity. He may reason something like this: "Well, I know that I 
spend an hour and a half on various kinds of general paperwork and, besides that, I'm sure I 
have thirty minutes of interruptions every day when one of these people comes to me. So I 
would say about two hours a day." 

Of course we can't write "2" down here because we've already begun to work in 
percents on this row, so you have to do a quick mental calculation and say, "That would be 
about 25% of your time. Does that sound right?" 

Francis may say, "It's probably more than that let's call it 30%". 

So you put 30% down under general administration. Then you say to Francis, "Is 
there anything else you do? I believe you spend some of your time revising the card catalog, 
don't you?" 

Francis says "Yes, but not as much as I spend on administration or circulation, but 
iSwseems to be quite a lot. I'd say I spend about 20% of my time on the catalog work." 

So you put 20 under "Catlgng." At this point Francis may say, "Wait a minute. I 
have 60 and 20 and 30, maybe I shou.'.d change that 60 to 50. 

DON'T let him do this. It can cause problems if he adjusts his estimate, just in order 
to make the numbers add up to exactly 100%. We arc only interested in the relative size of 
these jobs - for example, here Circulation is twice as much as Administration and three times 
as much as Cataloging. 
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This information will be lost if you or Francis start adjusting the numbers, to make 
them come out to exactly 100%. What you should do, even though they now add up to 
1 10%, is say, "Well, is there anything else that seems to represent a significant part of your 
work? I don't mean something that takes a half an hour every couple of weeks, but 
something that comes up regularly." If you are lucky Francis will say "No, that's about it. 
These three things constitute most of what I do." 

These examples are admittedly wordy, but v/e hope we have gotten across to you the 
fact that this has to be done in a very non-threatening way. If you are a boss in your library, 
whenever you talk to your staff about how they spend their time, they may respond by 
thinking, "He is checking up on me. Do I look like I am slacking off? Am I being called on 
to account for how I spend my time?" Of course that's not what's happening here. 

A functional cost study is not intended to target any particular employee, or to make 
people go through some "audit procedure." It's just a necessary step in understanding how 
the salary part of the library budget contributes to the various services the library's patrons 
receive. 

However, because people may have this fear, it's important to keep a relaxed attitude 
and to go back and forth a couple of times. It may even be necessary to go back to people 
four or five days after they have gone through this exercise and say, "I'm about to mail the 
data, but I thought I'd take a last look at these time and budget figures here and see if there 
is anything we left out or anything that we seem to have grossly mis-estimated. " 



[Inserts omitted here are images of handwritten completed forms] 
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The above example corresponds to the lengthy discussion given here. Of course in this study 
you will bring the service of interest into column. (3), and you will not need to record people 
who do not work on it at all. So your example, for the same people, would look more like the 
following: 



[Insert omitted here shows the same form with irrelevant items omitted.] 
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3 Costs and Beneficial impacts of Library Services: Rutgers 
APLab 

Estimation of Annual Service Rendered. 

at 

Date Prepared by: 

1 . (If known from a recent study or report) 

Annual number of uses of this Service 

For 12 months period from to 

IF (1) is not available, can you report: 

2. Number of uses of this service , 

during the survey or study period from to 



Additional Data (If you answered (1) you may omit this information. 

We would like any supporting information for the extrapolation from a study period to an 
entire year. If not other information is available, we will assume that there are 40 "typical 
weeks" in an academic year. If your study period included some weeks that should not be 
regarded as "typical" please explain in the comments. 

If other services or tallies were maintained during the period cited in answering (2) above, 
AND you have information on the annual totals for those tallies, please note them here. 



Service 


Count During Survey 


Annual Count 



























COMMENTS: Please tell us anything else that will help us to arrive at a realistic estimate of 
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how many times the service being costed was used during the most recent year. If you have 
no data whatsoever, please give us your "best educated guess". 
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Sending it all back to us: 

Please return the forms, as indicated in the drawing, to the Rutgers research team at the 
address shown. If you send forms for several different services in the same envelope, please 
be sure that they are clearly labelled and stapled into separate groups. 
Thank you for your cooperation. 
[Signatures] 




BEST COPY AVAILABLE 
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APPENDIX E 



Interviewer Training Manual 
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Study of Library Costs and Beneficial impacts 



INTERVIEWER TRAINING MANUAL 



Alexandria Project Laboratory 
School of Communication, Information 
and Library Studies 
Rutgers University 
November, 1993 



Adapted from The Eaqleton Poll Interviewer's Handbook , 
Eagleton Institute, Rutgers University. 

Joann D ' Esposito-Wachtmann 



APLAB-B%D\CLR\FINAL.REV\APPSC-H.51 Copyright (c) 1995 'Z Q J March 21, 1995 P. 101 



Rutgers Alexandria Project Lab Cost Benefit Study 1994 
Paul B. Kantor and Tefko Saracevic, Principal Investigators 

Appendix E 



SUMMARY 



When interviewing: 
Do 

A) Do try to make the respondent comfortable 
about answering the questions. 

B) Do read the questionnaire exactly as it is 
written . 

C) Do become familiar with the questionnaire. 

D) Do use neutral probes on open-ended 
questions . 

E) Do make sure your tape recorder is on during 
the interview - put one interview per side of 
the tape. 

F) Do remember to record the interview numbers 
on the tape and label the tape with the 
numbers . 

Do edit your work. 

Do use pencil to circle the response number. 



G) 
H) 

Don' t 



A) Don't bias the respondent's answers. 

B) Don't suggest answers to respondents. We 
want their ideas, not yours. 

C) Don't take "don't know" for an answer. Work 
at turning them into responses . 



er|c 
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INTRODUCTION 

The Alexandria Project Laboratory of the School of 
Communication, Information and Library Studies at Rutgers 
University has been awarded a grant from the Council on Library 
Resources to study library costs and benefits. 

The purpose of Phase I of the study is to explore why 
faculty members and students use the university library, how they 
use it, how this use is beneficial or not beneficial to them and 
how they measure this benefit or lack of benefit. 

Your role will be to gather this information via interviews 
with faculty and students who use particular services in one of 
five research libraries. Once you have collected this 
information, it will be analyzed by the project team with the 
ultimate goal of helping libraries to serve their users more 
effectively. 

The dialogue between the interviewer and the respondent is 
one of the most important parts of the study. It is at this 
point where the actual data is gathered so the research can meet 
the study objectives. For this reason, it is important that the 
information collected be of high quality, accurate and valid. To 
meet these objectives, it is crucial that interviewers: 

A) Develop a rapport with the respondent; 

B) Transmit the questionnaire as it is written; 

C) Know about the different types of questions; 

D) Probe until complete answers are obtained; 

E) Know how to handle a "don't' know" or 
"refusal" and when to terminate an interview; 
and 

F) Edit their work. 

We will discuss each of these points individually. 
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DEVELOPING RAPPORT 

At first glance, interviewing seems like a simple task. 
However, communication is more complex than it often appears. 
Individuals bring to the respondent- interviewer relationship 
their own personalities and experiences. A skillful interviewer 
is aware of the sensitivity needed in communicating with 
different types of respondents. An interviewer should be able to 
identify these differences and be an understanding listener to 
the opinions of others. 

The first step in the interviewing process is the 
development of a relationship with the respondent. The following 
descriptions are examples of some types of respondents and 
suggestions on how an interviewer may improve the interviewing 
relationship when these types are encountered. 

Types of Respondents 

The standard respondent is the type of person who makes up 
the largest portion of the people surveyed. This type of 
respondent will answer the interviewer's questions with little 
hesitation or explanation. The interviewer presents the 
questionnaire in a pleasant conversational manner and the 
respondent replies. Usually this type of respondent thinks of 
the interviewing experience as being very positive. As long as 
the interviewer reinforces this attitude the interview will flow 
smoothly . 

The reluctant respondent may be hesitant about completing 
the interview for several reasons. He/she may be suspicious of 
the motives for conducting the interview, may be in a hurry, or 
wish to maintain his/her privacy. Most people will give an 
interviewer an opportunity to talk. The interviewer can use this 
time to convince the reluctant respondent. If the standard 
introduction on the questionnaire does :.iot overcome the 
respondent's doubts, the interviewer may use his/her judgment and 
give the respondent additional information such as explaining the 
purpose of the study, or guaranteeing the respondent that the 
interview is confidential. The skillful interviewer can usually 
convince a reluctant respondent of the importance of the survey 
and can overcome any hesitations the respondent may have. 

The intimidated respondent feels he/she is inadequate. 
He/she does not think he/she has enough knowledge to answer the 
interviewer's questions. This respondent perceives the 
questionnaire as a test and may refuse the interviewer because 
he/she does not want to fail. If the interviewer is sympathetic 
and understanding of the respondent's feelings, the respondent 
will react favorably. Explain that the questions are being asked 
to find out people's opinions. Every opinion is valuable to the 
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survey and there are no right or wrong responses. The 
interviewer can relax the intimidated respondent by creating a 
warm, friendly atmosphere. 

The verbose respondent has opinions on everything. He/she 
has found someone to listen and is going to make sure he/she 
tells you everything he/she knows. The interviewer should be 
courteous, yet move the respondent on through the questionnaire. 
Statements like "That's interesting, let's go on to some other 
topics" or "Now, I'd like to find out what your opinions are on 
some other issues" can help the interviewer proceed through the 
questionnaire without alienating the respondent. 

Maintaining Rapport 

After the interviewer has established a pleasant 
relationship with the respondent, this atmosphere must be 
maintained. Even the most cooperative respondent will not feel 
comfortable with an interviewer who thoughtlessly races through 
the questionnaire or tediously drags the respondent from question 
to question. There is no general rule about the pace of an 
interview. In each interviewing relationship, the interviewer 
needs to determine what pace is most effective for that 
respondent. In some cases the respondent may give tin; 
interviewer an obvious cue and say, "I have to leave in a few 
minutes". On the other hand, sometimes just the respondent's 
manner will be the cue. For example, a very elderly respondent 
who has difficulty hearing and/or understanding may not verbalize 
this problem to the interviewer, but a skillful interviewer will 
be sensitive to this need and thoughtfully slow down the pace of 
the interview. 

A skillful interviewer will give the respondent the feeling 
that he/she is involved in a conversation with someone who values 
his opinions. The movement from question to question should be 
natural. If the interviewer cuts off the respondent in the 
middle of a response or daydreams between questions, the 
respondent will feel uncomfortable and the pleasant atmosphere of 
the interview deteriorates. Approach the interviewing process as 
you would a conversation with someone who is very interesting. 
Using your voice, make the respondent feel important. 
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TRANSMITTING THE QUESTIONNAIRE 

Although a sound relationship with the respondent is crucial 
to the interview process, it is also important that the 
interviewer skillfully execute the questionnaire. 

Verbatim Questioning 

The most effective way to transmit a questionnaire is 
through the technique of verbatim questioning. 

Verbatim questioning means: 

A) Reading the questions word for word as they 
are written . 

B) Asking the questions in the necessary order. 

Let us begin with reading the questions word for word. 

Questions asked in different ways elicit different 
responses. When the project directors go to analyze the data you 
collect, they will be grouping together all of the responses to a 
particular question and drawing conclusions from them. If the 
questions are not asked in a standard way, the conclusions are 
invalid. This is why we use verbatim questioning. 

To illustrate the point, examine the following questions. 
Notice how the meaning can change by slightly altering the 
wording. 

When was the last time you or someone in your household 
used the library? 

When was the last time you used the library to obtain 
materials for yourself or someone in your household? 

When was the last time you or someone in your household 
brought home something from the library? 

While each of these questions may look quite similar, they 
are in fact quite different in meaning. 

The second aspect of verbatim questioning is asking 
questions in the order they are written. This is because 
questions are ordered in a particular way for specific reasons; 
more general questions precede more specific ones, easier 
questions precede difficult ones, and occasionally, the response 
to one question determines whether the subsequent question is 
asked or skipped. (This is called a skip pattern and will be 
explained below) . 
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Thus, if the interviewer varies the order in which the 
questions are asked, the response to one question may bias or 
alter the response to another question and the standardization 
from one interview to another is lost. 

Enunciating/Familiarity with the Questionnaire 

Since the questions are being read to the respondent it is 
essential that the interviewer enunciate each word so the 
respondent does not have any difficulty understanding the 
question. Yet, the question should sound natural and 
conversational. If an interviewer stumbles over words, the 
respondent is affected and the atmosphere of the interview is 
threatened. Reading the questionnaire out loud several times 
before using it for an interview helps reduce errors that occur 
when an interviewer is unfamiliar with the questions he/she is 
asking the respondent . 

The Introduction 

It is important that you go right from the introduction to 
the first question. Asking permission could lead to an increase 
in refusals. Once the respondent begins to answer questions, the 
interview is in progress and the possibility of refusal is 
decreased. 
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TYPES OF QUESTIONS 

The questionnaire is composed of various types of questions, 
as outlined below: 

Closed/Open Ended Questions 

The questionnaire has two types of questions: open-ended 
and closed-ended. In a closed-ended question, the respondent 
must choose from one of the pre-designated answers. Usually, the 
response choices are written into the question. If you are not 
positive which answer the respondent is choosing, repeat all the 
choices. An example of a closed-ended question is: 

■How often do you use service name during a 
regular semester--less than once a month, 
once a month, 2-3 times a month, 4 times a 
month, or more than 4 times a month? 

Less than once a month 1 

Once a month 2 

2-3 times a month 3 

4 times a month 4 

More than 4 times a month. .5 
No response 0 

Another type of closed-ended question is the rating scale in 
which the respondent must choose the number on the scale which 
most closely reflects his/her opinion, such as: 

Using a scale from 1 to 7 where 1 means not 
at all familiar and 7 means very familiar, 
how would you rate your familiarity with this 
library? 

not at all very no 

familiar familiar response 

1 2 345670 

Even though you are tape-recording the interview, responses 
to closed-end questions must be recorded by CIRCLING IN PENCIL 
the number which corresponds to the response given. 

In open-ended questions, there are no preselected responses. 
The respondent gives whatever answer comes to his/her mind. this 
type of question tests the interviewers skills and requires 
PROBING, either the probes provided on the questionnaire, or one 
of the neutral probes discussed below. 

You need not write the answers to the open-ended questions 
on the questionnaire. The taped responses will be analyzed 
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later, by project staff, and converted to a numerical code which 
will then be written on the questionnaire. 

IT IS VERY IMPORTANT that the tape recorder be turned on, 
that the volume is up, that you have verbally recorded the 
interviewer number on the tape and that you get responses to most 
of the open-ended questions. 

Another type of open-ended question is the kind which 
requires the respondent to volunteer a numerical response. An 
example follows: 

About how much time did you spend using the 
reference service? 

minutes/hours no response 

12 0 

In this case the interviewer records the number on the blank 
line, then circles the number corresponding to the unit of time 
used by the respondent. 

Skip Patterns 

A skip pattern consists of a series of questions in which 
the respondent's answer to the initial question determines which 
question will be asked next. For example: 

1. If the reference services were not available 
at this library, is there something else you 
could have done to get this same benefit? 

Yes. (CONTINUE TO Q.2) 1 

No. (SKIP TO Q.3) 2 

No response. (SKIP TO Q.3).. 3 

2. What would you have done? (PROBE FOR 
SPECIFICS) 

3. If 100 represents the best possible 
experience with the reference service that 
you could imagine and 0 represents the worst, 
what score would you give for the way it 
actually turned out? 

In the example, if the respondent answers "yes" to Question 
1, he/she is asked Question 2 and then Question 3. If, however, 
the respondent answers "no" or does not answer Question 1, he/she 
skips to Question 3. 

It is very important to study the questionnaire and become 
familiar with the skip patterns before you begin interviewing. 
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Frequently, interviewers get so involved in a skip pattern they 
forget to record the answer to the first question. Be sure to 
circle the number of the response to the first question before 
you skip to the nxt one. 



9 7 Q 
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PROBING 

There are times during the interview when the interviewer 
needs to probe to obtain more information or more complete 
information. Probes have the following functions: 

1) They expand the information the respondent 
has given. 

2) They obtain specific information on a broad 
subject . 

3) They clarify what the respondent has said. 

The idea behind probing is TO GET AT THE MOST SPECIFIC ANSWERS 
POSSIBLE - TO CLARIFY WORDS UNTIL YOU CAN'T GET ANY MORE 
SPECIFIC . 

All probes must be NEUTRAL and NON-BIASING. 

The following situations make probes necessary: 

1) Respondents have difficulty putting thoughts 
into words . 

2) Respondents answers may be unclear or 
incomplete . 

3) Respondents feel insecure about giving a 
" wrong " answer . 



Types of Probes 

1 . The neutral question 

What do you mean by ? 

What e-i'se can you tell me? 

I'm not sure I understand what you have in 
mind. Could you tell me more about thai.? 
Is there anything else? 

2 . An expectant pause 

Some respondents need time to formulate an answer. 
Simply pausing gives them time to formulate their 
thoughts. However, you need a sensitivity to your 
respondent or a "pregnant pause" may become a "deadly 
silence" . 



Repeating the question 

A respondent who does not understand the question at 
first may realize its purpose when it is read a second 
time . 




An intonation of interest 

Respondents who feel the interviewer is truly 
interested in their opinions will be more willing to 
express their thoughts. Interviewers can accomplish 
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this by brief comments such as "Uh-huh" or "I see" or a 
simple nod. Caution: Any comments must be totally 
neutral. If you show approval or disapproval such as 
"Boy, do I agree with you", the respondents will 
attempt to give responses that obtain your approval 
even though they may not accurately express their true 
feelings . 
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NO RESPONSE/TERMINATIONS 

When a respondent replies "I don't know" it may mean any of 
the following: 

1) The respondent really does not know. The 
respondent does not feel he/she has enough 
information to answer the question. 

2) The respondent does not understand what the 
question has asked. 

3) The respondent feels he/she needs time to 
think, but also feels he/she must fill in the 
silence so he/she responds "I don't know". 

4) The respondent feels threatened that the 
interview is a test and he/she may give the 
wrong answer . 

Interviewers can often obtain answers from "don't know" 
respondents. A pause, a reassuring remark ("We're only 
interested in your ideas on this") or a neutral question ("What 
are your ideas about this?") can encourage a response. 

If a respondent refuses to answer a particular question but 
is willing to answer the majority of the questions, record the 
"no response" option. If, however, the respondent refuses to 
answer most of the questions, terminate the interview, and 
consider it as an incomplete on the Tally Sheet. 

Sometimes, because of a language problem, a hearing problem, 
or some other reason, it is necessary to terminate an interview 
(or not initiate one) . As an interviewer, you must use your 
judgment in determining when this is appropriate. When you do, 
tally it as an Interviewer Termination on the Tally Sheet. 

ALWAYS use pencil when writing on the questionnaire . 

EDITING 

When the interview is complete, the interviewer needs to 
check his/her work to make sure the interview has been filled out 
completely and accurately and that the open-ended questions were, 
in fact, recorded on the tape. It is important to do this 
IMMEDIATELY after the interview has ended, while it is still 
fresh in one's mind. 
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APPENDIX F 
Miscellaneous Forms 



F. 1 . Posting to recruit interviewers 

F.2. Letter to invite applicant 

F.3. Application form Wave 1 

F.4. Application form Wave 2 

F.5. Letter of appointment 

F.6. Interviewing Schedule 

F.7. Interviewer Tally Sheet 

F.8. Telephone interview Tally Sheet 

F.9. Letter at end of project 

F.10. Telephone interview schedule 

F. 1 1 . Call disposition form - Wave 2 

F.12. Site Feasibility Report 

F.13. Interviewer time sheet 

F.14. Examples of contact memos or messages 




a 

ERIC 
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[Prepare on organization letterhead] 

INTERVIEWERS WANTED FOR 
LIBRARY USAGE STUDY 
The Alexandria Project Laboratory at Rutgers University School of Communication, 
Information and Library Studies is seeking individuals interested in participating in a library 
usage study. 

The study will be conducted in two phases over the 1993-1994 academic year. Training 
will be provided. 




QUALIFICATIONS: 

-Willingness to approach library users and interview them using a pie- 
questionnaire. 

-Clear, spoken and written English. 
-Commitment of approximately 10-15 hours per week 
during the interviewing period (March 28-April 30) 
-Willingness to attend (paid) training sessions 
in New Brunswick. 

-Willing to work in one of the following cities: 
Manhattan 
New Brunswick 
Princeton 
Philadelphia 

Some knowledge of the language used by libra'ies, 
librarians, and library users to describe their 
principal operations and activities. 



designed 



COMPENSATION: $10.00/hr. 



Please respond as soon as possible to: 

Joann D'Esposito-Wachtmann (908) 932-7705 or 
E-MAIL: D@ZODIAC.RUTGERS.EDU. 



The university is an equal opportunity affirmative action employer, and qualified candidates 
will be reviewed without regard to race, creed, color, sex, religion, age, marital status, 
national origin, handicapped, status as a Vietnam veteran or disabled veteran, membership or 
non-membership in the union, or sexual preference. 
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[Prepare on organization letterhead] 
February 1994 



Dear Student, 

Thank you for your interest in the interviewer position. 

Please complete the enclosed application form and return to: 

Joann D'Esposito-Wachtmann 
Project Manager 
Alexandria Project Laboratory 
SCILS 

Rutgers University 

4 Huntington Street, Room 214 

New Brunswick, NJ 08903 

Thank you, 



Joann D'Esposito-Wachtmann 



o 

<Z ( J 
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APPLICATION 

Date 



Name 

Address. 



Phone ( ) 

Social security number. 



Are you : A citizen or national of the U.S.? 

A lawful permanent resident? 

(Alien # A ) 

An alien authorized to work until / / 

(Alien # or Admission # ) 

Proof of citizenship or immigration status will be required upon 
employment . 

Do you have interviewing experience? Yes No 

If yes, please explain. 



EDUCATION-Begin with most recent, include high school. 

. School 

Address 

Dates Attended 

Degree Received (if applicable) 

Major subject 

School 

Addr e s s 

Dates Attended 

Degree Received (if applicable) . 

Ma j or sub j ec t 

School 

Address 

Dates Attended 

Degree Received (if applicable) 

Major subject 

REFERENCES-Please provide the name and telephone number of two 
personal references (not relatives). 

Name 

Telephone Number 

Relationship 

Name 
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Telephone Number 

Relationship 

WORK EXPERIENCE-Begin with most recent. 

Company/ Institution 

Address 



Phone 

Dates employed 

Primary duties ^ 

Supervisor 

May we call supervisor to ask for a reference? Yes No 

Company/ Institution 

Addr e s s 



Phone : 

Dates employed 

Primary duties 

Supervisor 

May we call supervisor to ask for a reference? Yes No 

Company/ Institution 

Addr e s s 



Phone 

Dates employed 

Primary duties 

Supervisor 

May we call supervisor to ask for a reference? Yes No 

If you are hired, you will be required to attend a paid training 
session in New Brunswick between March 21 and March 26, 1994. 
Which of the following sessions are you able to attend? (check as 
many as apply) 



Are you willing to work 10-15 hours/week during the following 
weeks: March 28 - April 30, 1994? 



Thursday, March 24, 1994 



7 : 00 p.m. - 9 : 00 p.m. 
10: 00 a.m. -12 : 00 p.m. 
7 : 00 p.m. - 9 : 00 p.m. 



_Friday, March 25, 1994 



Friday, March 25, 1994 



Yes 



No 
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What days and hours are you available? Your schedule will be 
based on the information you provide here. Please be as precise 
as possible. 

Mw 



What is the maximum number of hours you are willing to work per 
week? 

hours 



What is the minimum number of hours you are willing to work per 
week? 

hours 



Please mark the city where you would prefer to work: 

Manhattan 

New Brunswick 

Princeton 



Please write (in your own handwriting) one or two paragraphs 
about why you are interested in this position and why you think 
you would be a good interviewer. 



I attest that all of the information I have provided is true to 
the best of my knowledge. 

S IGNATURE 

DATE 
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FOR OFFICE USE ONLY 

Interview yes 

Date : 



no 



Remarks. 



Employed 



.yes 



.no Date of employment. 



:RIC 
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[Prepare on organization letterhead] 

October , 1993 

Dear 

I am happy to offer you an interviewer position on the Alexandria 
Project Laboratory Cost /Benefit Study. As we have discussed, 
your work will run from November 15 through December i?5, 
excluding Thanksgiving week (November 21-27). In addition, you 
are required to attend a paid training session on November 13, 
1993 from 10:00 am - 12:00 pm in New Brunswick. You will be 
notified of your interviewing schedule and location closer to the 
onset of the interviewing period. The pay rate for this position 
is $10.00/hour. 

As an interviewer, you will be responsible for interviewing 
library users, either in person or on the telephone, about their 
use of library services and producing a specified number of 
neatly completed questionnaires with their accompanying tape 
recordings. You will report directly to me, the Project Manager. 

As representatives of Rutgers University and the School of 
Communication, Information and Library Studies all interviewers 
a:;e expected to comport themselves in a prof essional , courteous 
manner. Interviewers conducting face- to- face interviews are 
required to dress neatly and wear a name-tag at all times. 

For those interviewers traveling to New York or Princeton, 
transportation costs and travel-time from your place of residence 
to the interviewing site will be reimbursed. All interviewers 
can expect to be paid three to four weeks after all the 
interviewing is completed. 

Please find enclosed a copy of this letter for you to sign and 
return to me by October 22, 1993 as an acceptance of this 
position. 

Again, I am delighted to have you join our team. I believe we 
have a very solid group of interviewers who will help us provide 
valuable information to the library field and I look forward to 
working with you. 

Sincerely, 



Joann D ' Esposito-Wachtmann 

Project Manager 

JD/lr 

cc : Paul Kantor 

Tefko Saracevic 
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November 10, 1993 

To: #3 

From: Joann D ' Esposito-Wachtmann 

Subject: Interviewing Assignment/Schedule 



Presented below is your interviewing schedule for the week of 
November 15-20. 

IN- PERSON 



Location : 
Contact : 
Date/Time : 



********* Psychology Library 
[person's name] 
Tuesday, November 16, 1993 
1: 00 - 5 : 00 p.m. 

Thursday, November 18, 1993 
1:00 - 5:00 p.m. 



TELEPHONE 



Location: SCILS (Rutgers University) , Rcom 214 

Date/Time: Wednesday, November 17, 1993 

9:00 a.m. - 12:00 noon (if necessary) 



Tentatively, your schedule for the weeks of November 29 - 
December 3 and December 6-10 will be similar to the one above, 
However, I will confirm this after the first week of 
interviewing has been completed. 
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Tally Sheet 

Service 

Interviewer 

Date 

Hours 

Worked 



I 

COMPLETE 


Total 


INELIGIBLE RESPONDENT 


Total 


INCOMPLETE 


Total 


INTERVIEWER TERMINATE 


Total 


REFUSAL 


Total 


Total 



USE ONE TALLY SHEET PER SERVICE PER DAY 



o u 
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Date 

ADDRESS 

Dear 

On behalf of the CLR study project directors and myself, 
I would like to thank you for a job well done on the library 
usage study last semester. 

As I may have mentioned, this Spring we will be 
conducting Wave II of the same study. It is similar to the 
first wave in that we will interview library users on-site. 
In addition, two weeks after the in-person interview, we will 
telephone the respondents to determine the long-term impact of 
their library usage. 

If you would like to participate in this portion of the 
study, please advise me by February 5, as I will begin 
interviewing candidates soon after that. 

Again, thank you for your good work last semester. I 
loc'c forward to hearing from you. 

Sincerely, 



Joann D' Esposito-Wachtmann 
Project Manager 



C "■ '') 
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Wave II - CLR Study 
TELEPHONE INTERVIEWING SCHEDULE 
April 11-29, 1994 



Please call 932-7705 the day before you are scheduled to find 
out if it is necessary for you to come in. 



Friday 



Monday 



9-12 pm Christine 
Melissa 

Melissa 

Angela 

(backup) 

(backup) 



Tuesday 

Christine 
Andrea 
(backup) 



Wednesday 

Christine 
Andrea 
(backup) 



Thursday 

Christine 
Melissa 
(backup) 



1-5 pm 
Angela 



Angela 



Melissa 



Christine 
(backup) 



(backup) 



Angela 

Christine 
(backup) 



Melissa 

Christine 
(backup) 



Jennifer 

(1-2 pm) 
Angela 

(3-5 pm) 



6 -9pm 



Jennifer 
Christine 
(backup) 



Melissa 
Andrea 
(backup) 



Jennifer 
Melissa 
(backup) 



Melissa 
Christine 
(backup) 



If you are unable to work during your assigned hours, you are 
responsible for calling your backup and asking her to fill in 
for you. 
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Call Disposition Form 
Service 



Respondent's Name 

Respondent's Telephone Number 
Best Time to Reach 



Interview # (only for Completed Interview) 



Date 


Code 


Notes 


Inter 
viewer # 



































































































Codes 



Number No Longer Usable 

Completed Interview X 

Refusal R 
Non-Working/Incorrect # W 
Incomplete/Do not 

Recontact I 

Interviewer Terminate T 

Ineligible respondent E 
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No Answer/Busy N 
Answering Machine A 
(Do not leave a message) 
Incomplete/Recontact C 
(Indicate time in "Notes" 
column) 
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IMPACT STUDY FEASIBILITY REPORT 



Estimated Possible 
Approaches per hour 



Site : 

Function or Service: 
Observation: Date 



From 



to 



Sketch of the area: good interview posts; field of view 



Tally of Observed Users in Ranges: 




Estimated 
Coverage 



Discussions with: 



Notes on discussions: 



<; r. o 

l - 



o 
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Table CLRwlTOl 

Interviewer Time Sheet. Name 



Date 


Hours Worked 


TASK 


Number of Hours 


























































































































































































TOTAL 
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MATERIALS DELIVERY SERVICE SURVEY 
♦FACULTY AND GRADUATE STUDENTS* 

The Rutgers University School of Communications, Information and 
Library Studies (SCILS) is studying the impact of the Materials Delivery 
Service and would like to interview faculty and graduate students who have 
used the service. If you are willing to participate (15 to 20 minute 
interview) please send an e-mail message to dOzodiac or call 908-932-7705 
to volunteer. 



MATERIALS DELIVERY SERVICE SURVEY 
Faculty and Graduate Students 

The Rutgers University School of Communications, Information and 
Library Studies is studying the impact of the Materials Delivery Service 
and would like to interview you briefly. If you are willing to 
participate, please note your telephone number and/or email address here: 

Name : 

Campus or heme phone number: 

Best time to reach you: 

E-mail address: 



Faculty Graduate Student 

PLEASE LEAVE THIS ATTACHED TO THE REQUEST FORM IF YOU WANT TO 
VOLUNTEER . 
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The Rutgers University School cf Communication, Information and Library 

Studies is studying the impact of the online catalog 

and would like to interview you briefly by telephone. If you would like 
to participate, please provide the following information: 

Name 

Telephone number 

Best time to reach you by telephone 
E-mail address 

And please identify yourself as one of: 

Faculty, Staff, Graduate Student, Undergraduate Student, Other 
When finished, hold down control, then type X. 



The Rutgers University School of Communication, 
Information and Library Studies is studying the 
impact of the science document delivery service anr 1 
would like to interview you briefly by telephone. 
If you are willing to participate, please note 
the following information: 

Telephone Number: 

Best Time to Reach You By Telephone: 

E-mail Address: ; 

Please check one: 

Faculty/ Graduate 

Staff Student 

Undergraduate 

Student Other 

Thank You! 
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The Rutgers University School of Communication, 
Information and Library Studies is studying the 
impact of the science document delivery service and 
would like to interview you briefly by telephone. 
If you are willing to participate, please note 
the following information and mail this postcard. 

Name : 

Telephone Number: 

Best Time to Reach You: 

E-mail Address: 

Please check one: 

Faculty/ Graduate 

Staff Student 

Unde r g r adua t e 

Student Other 



Alexandria Project Laboratory 
School of Communication, Information 
and Library Studies 
Rutgers University 
4 Huntington Street, Room 214 
New Brunswick, NJ 08903 



Thank You! 




G 
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Focus Group Moderator Guide 
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Moderator Guide 
Cost/Benefit Study 
Wave I 

I. INTRODUCTION 

Hello. My name is Joann D'Esposito-Wachtmanu and I am the moderator 
today. 

The purpose of today's discussion is to gather opinions about library services. 
The session will take about an hour and a half. 

II. GROUND RULES 

1. Speak one at a time in a voice at least as loud as mine. 

2. Groups are being recorded because I need to report on the findings and that is 
the only way I'll remember. 

3. I'd like to hear from everyone but not everyone needs to answer every 
question. 

4. Please avoid side conversations with your neighbor because it interferes with 
the tape but feel free to address comments to one another. 

5. Asked you to come because we want your thoughts and opinions, both negative 
and positive. Negative comments won't hurt my feelings in any way. 

6. No smoking. 

7. As you know, there are no right or wrong answers in research. We're looking 
for different points of view so please have the courage to stick with your 
convictions -even if you arc the only one who feels u particular way. 

III. INTRODUCTION OF PARTICIPANTS 

I'm Joann. I am work at the School of Communication Information and 
Library Studies at Rutgers and I have a professional background in market 
research. 

0 r ) <; 
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Please tell us your name, department and how long you have been at the 
university. 
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IV. LIBRARY USE 



How frequently do you use the campus libraries? 

What services do you use the most?/How frequently. 

Thinking about the last time you used the library-either by physically going 
there, by sending someone or by computer, what service or services did you 
use? 

Generally, how experienced are you with that/those service(s)? 

What was the purpose of your interaction? 

1 . the purpose of the task 

2. the purpose of the information , if there was 
information involved 

How clearly defined was the problem which led you to use the service? 

Thinking about that one episode in particular, how much time did you spend using that 
service? 

V. BENEFITS OF LIBRARY USE 

What was the benefit, or the value, or the impact you got out of using this 
service? or What did you get out of using this service? How did that help 
you? 

Did it... 1. answer a question?/raise a question? 

2. refer you somewhere else? 

3. save time? 

4. save money? 

5. advance research? 

6. other? 

If no benefit, describe how it was not beneficial. 
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VI. MEASUREMENT OF BENEFIT 

For those who thought it was beneficial: 
What would you compare the benefit to? 
What dollar amount was it worth? 

Was the benefit you derived from using the service worth the amount of time 

you spent? How did you arrive at that conclusion? 

Was it convenient? What made it convenient? 

How else would you measure the benefit you derived from 
using the service? 



For those who thought it wasn't beneficial: 

What would you compare it to? 

Was it a waste of your time? 

Was it a waste of your money? 

Was it inconvenient? 

Was there somewhere else you could have gone to resolve the problem or 
accomplish this task? 

If yes, where would you have gone? 

How convenient would it have been? 

VII. RECAP 

This is what I've heard today. 

O u 
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VIII. ANYTHING ELSE? 



In this group, I have been trying to determine why you use library services, 
how you use them, if using them benefits you and how you measure this 
benefit. Is there anything else you think I should have asked that I didn't? 

IX. IF THERE IS TIME, ASK THEM THE QUESTIONNAIRE QUESTIONS AND SEE 
HOW THEY RESPOND. 
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APPENDIX H 



Questionnaire Instructions 



H.l. Wave 1 

H.2. Wave 2 Audition 

H.3. Wave 2 Callback 

H.4. Wave 2 Phone 

H.5. General Telephone instructions 

H.6. Site placement instructions 
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Rutgers Alexandria Project Cost/Benefit Study- 
Wave I 

Questionnaire Instructions 



Date 



Record the date, 



Interviewer # 



Interview # 



Time Started 

Coded 

Library- 
Service 
Version 



Put your interviewer 
number here . 

Number the questionnaires 
..consecutively. An 
incomplete will have a 
number, even though it is 
an incomplete. 

Record the time the 
interview began. 

Please disregard this. 

These are already f-^led in. 
Please disregard them. 



Introduction 



You may use your first 
name only, or your first 
and last names. Choose 
what makes you feel the 
most comfortable. 



Screening Questions 

(Some of you will have a screening question before p A ) 



Q.A 



The purpose of this 
question is to make sure 
we have the type of 
respondent we want. 

Our highest priority is 
faculty. If you have a 
choice between two 
potential respondents, 
approach the one you 
think might be a faculty 
member . 

If there are so many 
potential respondents 
that there is a 
possibility you will miss 
a faculty member because 
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you are' with a student, 
wait five minutes between 
interviews. Or, if less 
than five minutes has 
passed and you get a 
"student" response to 
Q.A, terminate and tally 
as an ineligible 
respondent . 

Q.B This question is here so 

that we don't interview 
the same person twice. 

If the person does not qualify, erase the responses, tally as an 
ineligible respondent on the Tally Sheet and re-use the 
questionnaire . 

If the person does qualify, this is the point where you turn on 
the tape recorder, record the interview number and, if necessary, 
ask the respondent to hold the tape recorder. IT IS EXTREMELY 
IMPORTANT THAT THE RESPONDENT'S VOICE IS CLEARLY AUDIBLE ON THE 
TAPE III 

If the respondent asks why you are taping the interview, respond 
with: 

I am able to execute the interview morp quickly because I need 
not write down your responses to the open-ended questions. 

Q.l Use your probing skills to 

find out why they needed 
that information or document. 

Q.2 Circle t'.ie appropriate 

number . I f the 
respondent gives a verbal 
response rather than a 
numerical one, re-read 
the question. 

Q.3 We want to know, in very 

specific terms, what their 
project or work is about 
and how it relates to their 
use of the service. 

Q.4 We are trying to determine 

how that service, piece of 
information, or document 
impacted the respondent's 
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project or work. 

Q.5 Circle the appropriate 

number. Then, follow the 
skip pattern 
instructions . 

Q's. 5a-5e are skip pattern questions. If the response to Q.5 
was 1 or 2 , go to Q. 5c. If the response to Q.5 was 3, 4, 5, 6, 
or 7, go to Q.5a. If there was no response to Q. 5, skip to Q. 
6. 

Q.5a Probe for specifics. 

Q.5b Probe for specifics. 

Q.5c Probe for specifics. 

Q.5d This is a skip pattern. 

If the response is "yes," 
continue to Q.5e. If the 
response is "no" or "no 
response," go to Q.6. 

Q.5e Probe for specifics. 

Q.6 Circle the appropriate number. 

Q.7 Probe for specifics. 

Q.8 Circle the appropriate number. 

Q.9 Write the response on the 

blank line, then circle the 
number below the appropriate 
unit of time. If the 
respondent gives you a range, 
probe to obtain only one 
figure . 

Q.10 Circle the appropriate number. 

Q.ll Write the dollar amount. Do 

not write cents. If the 
respondent gives you a range, 
probe to obtain only one 
figure . 

Often, respondents are hesitant 
to answer this question, remind 
them that we only want their 
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opinion. If necessary, reassure 
them that we are not asking the 
question so that the library can 
begin charging them for services. 

Q.12 Circle the appropriate 

number, then follow the 
skip pattern 
instructions. 

Q's. 12a-15 are skip pattern questions. Only ask them if the 
answer to -Q.12 was "yes". 

Q.12a Usually, the respondents will 

go right into the answer to 
Q.12a when they answer Q.12. 
If they do, it is alright to 
not ask Q.12a. THIS IS A 
MAJOR EXCEPTION. 

Circle the appropriate number. 

Write the number corresponding 
to the response, then circle 
the number below the appropriate 
unit of time. Do not accept 
ranges . 

Write in the dollar 
amount. Do not write 
cents . Do not accept 
ranges . 

Write one digit per line. This 
number should be between 0 and 100. 

Probe for specifics. 



Q.13 
Q.14 

Q. 15 

Q.16 
Q. 17 



Q.li 



Here we are referring to 
the library , not the 
service. Circle the 
appropriate number. 



Q.19 



Here we are referring to 
the service . Circle the 
appropriate number. 



Q.20 



Be sure to read the list 
of response options. 
Omit "no response." 
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Q.21 



If you are interviewing a 
student, ask the first 
question. If you are 
interviewing a faculty or 
staff member, ask the second 
question. Record the response 
on the blank line. Disregard 
the section with the 
" INTERVIEWER DISREGARD" heading. 



Q.23 



Be sure to read the list 
of response options. 
Omit "no response" . 



TURN OFF THE TAPE RECORDER. 



Gender 



Record the respondent ' s 
gender . 



Time ended 



Record the time the 
interview ended. 



Time began 



Copy the time the interview 
began from page 1 . 



Interview length 



Subtract and record the 
interview length. 
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Rutgers Alexandria Project Cost/Benefit Study 

Wave II 
Questionnaire Instructions 
AUDITION 



Date Record the date here. 

Interviewer # Put your interviewer 

number here . 

The questionnaires have 
been prenumbered. 

Time the interview began. 

Already filled in. 
Please disregard these. 



When you are finished 
with the interview you 
will circle the response 
to Q.3 here. 

You may use your first 
name only, or your first 
and last names. Choose 
what makes you feel the 
most comfortable. 

Screening Questions 

(Some of you will have a screening question before Q.A) 

Q.A The purpose of this 

question is to make sure 
we have the type of 
respondent we want . 

Our highest priority is 
faculty. If you have a 
choice between two 
potential respondents, go 
for the one you think 
might be a faculty 
member . 

If there are so many 
potential respondents 
that there is a 
possibility that you will 



Interview # 



Time Started 

Library 
Service 
Version 

Q.3 



Introduction 
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miss a faculty member 
because you are with a 
student, wait five 
minutes between 
interviews, or if less 
than five minutes have 
passed and you get a 
"student" response to 
Q.A, terminate and tally 
as an ineligible 
respondent . 

Q.B This question is here so 

that we don't interview 
the same person twice. 

If the person does not qualify, erase the responses, tally as an 
ineligible respondent on the Tally Sheet and re-use the 
questionnaire . 

If the person does qualify, this is the point where you turn on 
the tape recc.der, record the interview number and ask the 
respondent t. hold the tape recorder. IT IS EXTREMELY IMPORTANT 
THAT THE RESPONDENT'S VOICE IS CLEARLY AUDIBLE ON THE TAPE!!! 

If the respondent asks why you are taping the interview, respond 
with: 

"I am able to execute the interview more quickly because I need 
not write down your responses to the open-ended questions." 

Q.l Probe for specifics. 

Q.2 Circle the appropriate 

number . 

Q.3 Circle the appropriate 

number. Note the skip 
instructions for 
responses 3-7. 

Q.3a Only ask this question if 

the response to Q.3 was 
"1" , "2" or "-1" . 
Probe for specifics. 

Q.3b Circle the appropriate number and note 

the skip instructions. 

Q.3c Only ask this question if the 

answer to Q.3b was "yes". 
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Q.4 



Q.5 



This question has been omitted 
on some of the questionnaires. 
If it is included, circle the 
appr opr i a t e numbe r . 

This question has been 
omitted on some 
questionnaires. If it 
included ba sure to probe 
for specifics. 



Q.6 



Circle the appropriate number. 



Q.7 



Write the number corresponding to 
the response, then circle the 
number below the appropriate unit 
of time. DO NOT ACCEPT FRACTIONS 
OF AN HOUR (i.e. 1 1/2 hours) . 
CONVERT IT TO MINUTES (i.e. 90 
minutes) . 



We will be calling back the on-site respondents with follow-up 
questions and, therefore, we need to know their name, telephone 
number and the best time to reach them. Record this information, 
along with the interview number, on page 4. (Page 4 will 
eventually be detached from the rest of the questionnaire) . 

TURN OFF THE TAPE RECORDER AND GO BACK TO PAGE 3 . 



Gender 



Time ended 



Record the respondent's 
gender . 

Record the time the 
in t e r v i ew ended . 



Time began 



Copy the time the interview 
began from page 1 . 



Interview length 



Subtract and record the 
length. 



When we call back the respondents, we will only be calling those 
who gave responses 3-7 to Q.3. So that we can know who those 
people are at a glance, please mark the answer to Q.3 on page 1. 

Finally, when we call back the respondent, we will need to know 
their status (student, faculty, etc.) and remind them of the 
specific research question about which we are calling. On page 
4, record their status and write a brief Synopsis of the re search 
question or problem based upon the answer to Q.l so that the 
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interviewer conducting the call-back will be able to descrioe it. 
(Please see attached examples) . 
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Examples of Synopses 

1. Respondent was tracing the legislative history of the 
meals on wheels program, (on March 28, 1994) 

2. Respondent was researching the influence of the 
commedia dell 'arte on contemporary theater and 
improvisation theory, (on March 28, 1994) 

3 . Respondent was doing research for a college level 
anthology of 17th and 18th century American writing. 

(on March 28, 1994) 



31 i 

q APLAB-B%D\CLR\FINAL.RE\AAPPSC-H.51 Copyright (c) 1 995 March 21, 1995 P. 154 

ERIC 

UMMimmmM 



Rutgers Alexandria Project Lab Cost Benefit Study 1994 
Paul B. Kantor and Tefko Saracevic, Principal Investigators 

Appendix H 



Rutgers Alexandria Project Cost/Benefit Study 

Wave II 
Questionnaire Instructions 
CALLBACK 



Date 

Interviewer # 
Interview # 

Time Started 

Library 
Service 
Version 

Introduction 

Q.la 
Q.lb 
Q.2 

Q.2a 

Q.2b 

Q.3 
Q.4 

Q.5 



Record the date here. 

Put your interviewer 
number here. 

Copy the questionnaire 
number from the audition 
sheet . 

Time the interview began. 

Already filled in. 
Please disregard this. 



To describe the project, 
use the synopses from the 
audition. 

Probe for specifics. 

Probe for specifics. 

Circle the appropriate number. 
Note the skip instructions. 

Only ask this question if the 
response to Q.2 was "3", "4", "5", 
"6", or "7". Probe for specifics. 

Only ask this question if the 
response to Q.2 was "3", "4", "5", 
"6", or "7". Probe for specifics. 

Circle the appropriate number. 

Circle the appropriate number. 
Note the skip instructions. 

Circle the appropriate number. 
Only ask this question if the 
answer to Q.4 was' "Has had an 
impact " . 
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Q.6 



Write in the dollar amount. DO NOT 
WRITE CENTS. IF THE RESPONDENT 
GIVES YOU A RESPONSE WITH CENTS, 
ASK THEM FOR A WHOLE NUMBER. 



Sometimes , 
hesitant to 
question . 
we only wan 
waiting a c 
usually ill 
Try this te 
accepting a 



respondents are 

answer this 
Reminding them that 
t their opinion and 
ouple of seconds 
icits a response, 
chnique before 
"no response" . 



Q.7 



Q.8 



Q.9 



Q.10 



The respondent could have 
priced it anywhere else. 
Circle the appropriate 
response . 

We are no longer referring to 
the specific library or 
service in this question. The 
respondent could have 
purchased any information 
services anywhere in 
connection with this project. 
Note the skip instructions. 

Only ask this question if the 
response to Q.8 was "yes". 
Read each item on the list and 
record a response for each 
item. For "some other funds" 
write the response in the 
space provided. Accept as 
many "yes" responses as the 
respondent gives you. 

Only ask these questions to 
faculty/staff. Read uhe list 
of services and record the 
response in the Q.10 column by 
circling the appropriate 
number. Do not read 
"libraries". (We already know 
they have used the library 
because that is where we 
interviewed them initially) . 
For "some other services" 
write the response in the 
space provided. 
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Q. 11 



Only ask this question to 
faculty/staff. For each "yes' 
item, in Q.10, ask Q.ll. 
Repeat the scale: definitions 
when necessary. Circle the 
appropriate number. 



Q. 12 



Ask students "What is your area of 

study?" Ask faculty "with what 

department are you affiliated?" 

All others, skip to Q.13. 

RECORD THE RESPONSE ON THE 

LINE. Disregard the 

" INTERVI EWER DI SREGARD " 

section . 



Q.13 



Circle the appropriate 
number . 



Time Ended 



Record the time the interview 
ended . 



Time Began 



Record the time the 
interview began from page 
1. 



Interview Length 



Subtract and record the 
length . 
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Rutgers Alexandria Project Cost/Benefit Study- 
Wave II 
Questionnaire Instructions 
PHONE 



Date Record the date here. 

Interviewer # Put your interviewer 

number here . 

The questionnaires have been 
prenumbered . 

Time the interview began. 

Already filled in. 
Please disregard these. 



You may use your first 
name only, or your first 
and last names. Choose 
what makes you feel the 
most comfortable. 

We are only interviewing faculty and 
graduate students. 



This question is here so 
that we don't interview 
the same person twice. 

If the person does not qualify/ erase the responses, tally as an 
ineligible respondent on the Call Disposition Form and re-use the 
questionnaire . 

If the person does qualify, this is the point where you turn on 
the tape recorder and record the interview number. 

If the respondent asks why you are taping the interview, respond 
with: 



"I am able to execute the interview more quickly because I need 
not write down your responses to the open-ended questions." 
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Q.l 
Q.2 

Q. 3a 
Q.3b 
Q 4 

Q. 4a 

Q. 4b 

Q.4c 

Q.4d 
Q.4e 



Probe for specifics. 

Circle the appropriate 
number . 

Probe for specifics. 

Probe for specifics. 

Circle the appropriate number. 
Note the skip instructions. 

Only ask this question if the 
response to Q.4 was "3", "4", "5", 
" 6 " or " 7 " . 

Only ask this question if the response 
to Q.4 was " 3 " , " 4 " , " 5 " , "6 " or " 7 " . 
Skip to the next page when finished. 

Only ask this question if the 
response to Q.4 is "1", "2", or 
Probe for specifics. 



Circle the appropriate response, 
the skip instructions. 

Only ask this question if the 
response to Q.4e is "yes". 



Note 



Q.5 



Q. 5a 



This question has been 
omitted on some 
questionnaires. If it 
included, circle the 
appropriate number. 

This question has been omitted on 
some questionnaires. If it is 
included, be sure to probe for 
specifics . 



Q.6 
Q-7 



Circle the appropriate number. 

Write the number corresponding to 
the response, then circle the 
number below the appropriate unit 
of time. DO NOT ACCEPT FRACTIONS 
OF AN HOUR (i.e. 1 1/2 hours). 
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CONVERT IT TO MINUTES (i.e.. 90 
minutes) . 



Q.8 Circle the appropriate number. 

Q.9 Circle the appropriate number. Note the 

skip instructions. 

Q.10 Circle the appropriate number. 

Q.ll Write in the dollar amount. DO NOT 



WRITE CENTS. IF THE RESPONDENT 
GIVES YOU A RESPONSE WITH CENTS , 
ASK THEM FOR A WHOLE NUMBER. 

Sometimes, respondents are hesitant 
to answer this question. Reminding 
them that we only want their 
opinion and waiting a couple of 
seconds usually illicits a 
response. Try this technique 
before accepting a "no response" . 



Q.12 The respondent could have 

priced it anywhere else. 
Circle the appropriate 
response. 

Q.13 We are no longer referring to the 

specific library or service in this 
question. The respondent could have 
purchased any information services 
anywhere in connection with this 
project. Note the skip instructions. 

Q.14 Only ask this question if the response 

to Q.13 was "yes". Read each item on 
the list and record a response for each 
item. For "some other funds" write the 
response in the space provided. Accept 
as many "yes" responses as apply. 

Q.15 Only ask this question to 

faculty/staff. Read the list of 
services and record the response in 
the Q.15 column by circling the 
appropriate number. Do not read 
"libraries". (We already know they 
have used the library because that 
is where they volunteered to 
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participate). For "some other 
service" , write the response in the 
space provided. 



Q. 16 



Only ask this question to 
faculty/staff. For each "yes" item 
in Q.10, ask Q.ll. Repeat the 
scale definitions when necessary. 
Circle the appropriate number. 



Q. 17 



Ask students "What is your 
area of study?" Ask faculty 
"with what department are you 
affiliated?" RECORD THE 
RESPONSE ON THE LINE. 
Disregard the "INTERVIEWER 
DISREGARD" section. 



Q. 18 



Circle the appropriate number 



Gender 



Record the respondent ' s 
gender . 



Time ended 



Record the time the 
interview ended. 



Time began 



Copy the time the interview 
began from page 1 . 



Interview length 



Subtract and record 
the length. 
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Telephone Interviewing Instructions 



1. Note the "Best Time to Reach" on the Disposition Form 
and call the respondent. 

2. Record the date, the code for the- outcome of the call, 
any relevant notes and your interviewer number on the 
Call Disposition Form. 

3. FOR CALLBACKS: If the call resulted in a completed 
interview, copy the Interview Number from the Audition 
onto the Questionnaire and onto the Call Disposition 
Form. 

FOR PHONE: If the call resulted in a completed 
interview, write the Interview Number on the Call 
Disposition Form. 

4. Paper clip the Call Disposition Form to the completed 
Interview and leave it and the corresponding tape in 
the box marked "complete" on the window ledge. 

5 . Remember to write the Interview Number on the tape case 
and on the tape . 

6 . Do Not leave messages on answering machines . 

7. If you are asked "Where did you get my number?", remind 
the respondent that they volunteered either by filling 
out a form or by sending us an electronic message 
saying they would participate. 
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Telephone Interviewing Instructions 



1. Write the date, respondent's name and respondent's 
telephone number on the Call Disposition form. 

2. Make the call. 

3. If the interview is completed, put the "X"code in the 
"Code" column on the Call Disposition form. If not, 
put the appropriate code. 

4. If we need to recontact the person at a specific time, 
write the time in the "Notes" column. 

5. Interview numbers will all begin with the number 9 
followed by two digits beginning with 01 and continuing 
on consecutively. Write the interview # in the "#" 
column. 

6. Write your interviewer number in the "Int" column on 
the Call Disposition form. 

7. Write the service name (from p.l - top left on the 
questionnaire) . 

8. DO NOT LEAVE MESSAGES ON ANSWERING MACHINES. 

9. If you are asked: "Where did you get my number?" 
Respond: 

We are working with the library director's office 
to study the impact of the library's services and 
they suggested you would be a good person to call. 

If we have an agreement form from them, please remind 
them that they filled out a form on which they agreed 
to be contacted. 
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Example Special Instructions 
Biology Library 

1. Please introduce yourself to upon 

your arrival. He will show you the best place to 
position yourself. 

2 . We are interviewing faculty and graduate students 
who have used one of the online services. 

3. Your total quota is 33 interviews. Try to obtain as 
many as possible during each shift. 

4. Please bring the completed questionnaires to SCILS 
Room '214 the day after your shift. 

This is the end of file appsc-h. 51 jps of the Rutgers APLab 
report on the Cost .Value . Study . 
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